How to Copy HDFS files to Local Linux GET Vs copyToLocal

Copy HDFS files to local Linux, which you can do in two ways. Those are GET and copyToLocal. There is a little difference between these two.

How to get File from Hadoop to Local

Either you can use GET or copyToLocal command to copy files to local from HDFS. Check here how it is.

1. The Get Command

GET Command

The get command copies HDFS-based files to the local Linux file system. The get command is similar to copyToLocal, except that copyToLocal must copy to a local Linux file system based file.

[hadoop@hc1nn tmp]$ hdfs dfs -get /tmp/flume/agent2.cfg
#Display the list of files
[hadoop@hc1nn tmp]$ ls -l ./agent2.cfg
-rwxr-xr-x. 1 hadoop hadoop 1343 Jul 26 20:23 ./agent2.cfg

This example copies the HDFS-based file agent2.cfg to the local Linux directory (” . “).

Take away

  1. copyToLocal, which is file to file of Linux
  2. GET command: You can use to copy HDFS files to local Linux directory.

2. The copyToLocal Command

copyToLocal Command

hadoop fs -copyToLocal [-ignorecrc] [-crc] URI <localdst>

Similar to get command, except that the destination is restricted to a local file reference.

Take Away

  1. When you have local file reference, in local LINUX, so you can copy the files from HDFS.

References

” Success is not final, failure is not fatal: it is the courage to continue that counts.”

— Anonymous

5 Distributed File System (HDFS) Top Features

Resource Sharing

The popularity of computer system arises due to nature of some applications. In such cases, it is necessary to facilitate sharing long-storage devices and their data to make system more user friendly.

Transparency

The main functionality of DFS is transparency which means user would be unaware about data location, movement, access, etc.

High Availability

The main feature of DFS is high availability. This feature states that if one server goes offline or failure, the data stored on its hard drives is still available for other nodes.

Location Independence

File name should not be changed when its physical location changes.

User Mobility

Access to file from anywhere or from any remote location.


LATEST POSTS

Read now all latest posts.

SQL Query: How to Use LOCATE and Substring Correctly

Here are SQL queries to use the SUBSTRING function along with LOCATE function. LOCATE gives position, and SUBSTRING uses that for the start position.

Python Logic to use tell() and seek()

In Python, both tell and seek functions help you get the data from a file. How to use these explained on how to use in your current project.

11 Kafka Admin Interview Questions

You can learn interview questions for the Kafka administrator role here. The questions covered prime points of the Kafka server.

Loading…

Something went wrong. Please refresh the page and/or try again.

Author: Srini

Experienced software developer. Skills in Development, Coding, Testing and Debugging. Good Data analytic skills (Data Warehousing and BI). Also skills in Mainframe.