Error while copying file to HDFS in hadoop ecosystem

While issuing a command to copy file from Local File system to HDFS on terminal in Hadoop 3.0, it shows error

hadoop-3.0.0/hadoop2_data/hdfs/datanode': No such file or directory:
`hdfs://localhost:9000/user/Amit/hadoop-3.0.0/hadoop2_data/hdfs/datanode.

However I have checked that the directory hadoop-3.0.0/hadoop2_data/hdfs/datanode exists with the proper access rights. I tried uploading file from Web browser and it shows following error.

"Couldn't find datanode to write file. Forbidden"

Please help to resolve the issue. Attaching the core-site.xml

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>

hdfs-site.xml

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/Amit/hadoop-3.0.0/hadoop2_data/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.name.dir</name>
<value>file:/home/Amit/hadoop-3.0.0/hadoop2_data/hdfs/datanode</value>
</property>
</configuration>

Checked the Datanode logfile in Hadoop Installation directory and it is showing success message as

INFO org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]
INFO org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker: Scheduling a check for [DISK]file:/tmp/hadoop-Amit/dfs/data
INFO org.apache.commons.beanutils.FluentPropertyBeanIntrospector: Error when creating PropertyDescriptor for public final void org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)! Ignoring this property.
INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started
INFO org.apache.hadoop.hdfs.server.common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
INFO org.apache.hadoop.hdfs.server.datanode.BlockScanner: Initialized block scanner with targetBytesPerSec 1048576
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Configured hostname is DESKTOP-JIUFBOR.localdomain
INFO org.apache.hadoop.hdfs.server.common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting DataNode with maxLockedMemory = 0
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened streaming server at /0.0.0.0:9866
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwidth is 10485760 bytes/s
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Number threads for balancing is 50
INFO org.eclipse.jetty.util.log: Logging initialized @146677ms
INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.datanode is not defined
INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode
INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 49833
INFO org.eclipse.jetty.server.Server: jetty-9.3.19.v20170502
INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@3a0baae5{/logs,file:///home/Amit/hadoop-3.0.0/logs/,AVAILABLE}
INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@289710d9{/static,file:///home/Amit/hadoop-3.0.0/share/hadoop/hdfs/webapps/static/,AVAILABLE}
INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.w.WebAppContext@3016fd5e{/,file:///home/Amit/hadoop-3.0.0/share/hadoop/hdfs/webapps/datanode/,AVAILABLE}{/datanode}
INFO org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@178213b{HTTP/1.1,[http/1.1]}{localhost:49833}
INFO org.eclipse.jetty.server.Server: Started @151790ms
INFO org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Listening HTTP traffic on /0.0.0.0:9864
INFO org.apache.hadoop.util.JvmPauseMonitor: Starting JVM pause monitor
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnUserName = Amit
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: supergroup = supergroup
INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 1000 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 9867
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server at /0.0.0.0:9867
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: null
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: <default>
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:9000 starting to offer service
INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9867: starting
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Acknowledging ACTIVE Namenode during handshakeBlock pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:9000
INFO org.apache.hadoop.hdfs.server.common.Storage: Using 1 threads to upgrade data directories (dfs.datanode.parallel.volumes.load.threads.num=1, dataDirs=1)
INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /tmp/hadoop-Amit/dfs/data/in_use.lock acquired by nodename
INFO org.apache.hadoop.hdfs.server.common.Storage: Analyzing storage directories for bpid BP-1751678544-127.0.1.1-1518974872649
INFO org.apache.hadoop.hdfs.server.common.Storage: Locking is disabled for /tmp/hadoop-Amit/dfs/data/current/BP-1751678544-127.0.1.1-1518974872649
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Setting up storage: nsid=1436602813;bpid=BP-1751678544-127.0.1.1-1518974872649;lv=-57;nsInfo=lv=-64;cid=CID-b7086125-1e01-4cf4-94d0-f8b6b1d4db25;nsid=1436602813;c=1518974872649;bpid=BP-1751678544-127.0.1.1-1518974872649;dnuuid=f132f3ae-7f95-424d-b4d0-729602fc80dd
INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added new volume: DS-ba9d49d2-87cb-4dff-ae80-d7f11382644f
INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added volume - [DISK]file:/tmp/hadoop-Amit/dfs/data, StorageType: DISK
INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Registered FSDatasetState MBean
INFO org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker: Scheduling a check for /tmp/hadoop-Amit/dfs/data
INFO org.apache.hadoop.hdfs.server.datanode.checker.DatasetVolumeChecker: Scheduled health check for volume /tmp/hadoop-Amit/dfs/data
INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-1751678544-127.0.1.1-1518974872649
INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-1751678544-127.0.1.1-1518974872649 on volume /tmp/hadoop-Amit/dfs/data...
INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-1751678544-127.0.1.1-1518974872649 on /tmp/hadoop-Amit/dfs/data: 1552ms
INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to scan all replicas for block pool BP-1751678544-127.0.1.1-1518974872649: 1597ms
INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding replicas to map for block pool BP-1751678544-127.0.1.1-1518974872649 on volume /tmp/hadoop-Amit/dfs/data...
INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice: Replica Cache file: /tmp/hadoop-Amit/dfs/data/current/BP-1751678544-127.0.1.1-1518974872649/current/replicas doesn't exist
INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time to add replicas to map for block pool BP-1751678544-127.0.1.1-1518974872649 on volume /tmp/hadoop-Amit/dfs/data: 1ms
INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to add all replicas to map: 4ms
INFO org.apache.hadoop.hdfs.server.datanode.VolumeScanner: VolumeScanner(/tmp/hadoop-Amit/dfs/data, DS-ba9d49d2-87cb-4dff-ae80-d7f11382644f): no suitable block pools found to scan. Waiting 1811581849 ms.
INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: Periodic Directory Tree Verification scan starting at 2/18/18 8:15 PM with interval of 21600000ms
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1751678544-127.0.1.1-1518974872649 (Datanode Uuid f132f3ae-7f95-424d-b4d0-729602fc80dd) service to localhost/127.0.0.1:9000 beginning handshake with NN
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool Block pool BP-1751678544-127.0.1.1-1518974872649 (Datanode Uuid f132f3ae-7f95-424d-b4d0-729602fc80dd) service to localhost/127.0.0.1:9000 successfully registered with NN
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode localhost/127.0.0.1:9000 using BLOCKREPORT_INTERVAL of 21600000msec CACHEREPORT_INTERVAL of 10000msec Initial delay: 0msec; heartBeatInterval=3000
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Successfully sent block report 0xe646383a22bd4be5, containing 1 storage report(s), of which we sent 1. The reports had 0 total blocks and used 1 RPC(s). This took 9 msec to generate and 834 msecs for RPC and NN processing. Got back one command: FinalizeCommand/5.

1 Answer

Executive summary:

It seems like you are trying to copy the files in non-existed HDFS folder

Detailed Answer:

There is a huge difference between HDFS file-system and regular file-system.

HDFS file-system - Hadoop Distributed File System (HDFS) is designed to reliably store very large files across machines in a large cluster.

The file system is distributed among several machines, and can be accessed only by HDFS commands (or equivalent).

Assuming that you were using /user/Amit/hadoop-3.0.0/hadoop2_data/hdfs/datanode as HDFS target folder - I suspect that the folder doesn't exists.

You can test my assumption when running the following commands:

• Copy file into HDFS /tmp folder

  hadoop fs -put <LocalFileSystem_Path> /tmp

• Copy file into HDFS default folder (.)

  hadoop fs -put <LocalFileSystem_Path> .

Afterwards you can perform the ls (list files) command - to see if the files are there:

• List files in HDFS /tmp folder

  hadoop dfs -ls /tmp

• List files in HDFS default folder (.)

  hadoop dfs -ls .

More info about - HDFS File system shell can be found here:

Update:

In order to check if the HDFS folder /user/Amit actually exists, you can execute the following command:

hadoop dfs -ls /user/Amit

If the folder exists, you might be able to copy files there using:

hadoop fs -put <LocalFileSystem_Path> /user/Amit

If the folder doesn't exists, you'll need to investigate the HDFS file system, using e.g.:

hadoop dfs -ls /

Followed by performing ls on the sub-directories.

If you have the relevant permissions, you will be able to create sub-folders, e.g. if the folder /user/Amit exists, you might be able to execute:

hdfs dfs -mkdir /user/Amit/newsubfolder

Note, you can try using the mkdir -p option, which will make also the parents folders (if you have the needed permissions) see two examples below (I guess that you have the needed permissions for the first command):

hdfs dfs -mkdir /tmp/Amit/fold1/fold2
hdfs dfs -mkdir /user/Amit/fold1/fold2