hdfs dfs -ls \/user\/justin\/dataset3\/<\/code><\/pre>\n\n\n\nLike Linux, other commands exist <\/a>for finding information about file and filesystem usage. For example, the du<\/code> command gives the amount of space used up by files in that directory:<\/p>\n\n\n\nhdfs dfs -du \/user\/justin\/dataset3\n1147227 part-m-00000<\/code><\/pre>\n\n\n\nand the df<\/code> command shows the capacity and free space of the filesystem<\/a>. The -h option displays the output in human-readable format, instead of a very long number.<\/p>\n\n\n\nhdfs dfs -df -h \/\nFilesystem Size Used Available Use%\nhdfs:\/\/local:8020 206.8 G 245.6 M 205.3 G 0%<\/code><\/pre>\n\n\n\nThe hdfs command also supports other filesystems<\/a>. You can use it to report on the local filesystem instead of hdfs, or other filesystems for which it has a suitable driver. Object storage systems such as Amazon S3<\/a> and Openstack Swift<\/a> are also supported, so you can do this:<\/p>\n\n\n\nhdfs dfs -ls file:\/\/var\/www\/html the local filesystem\nhdfs dfs -df -h s3:\/\/dataset4\/ an Amazon s3 bucket called dataset4.<\/code><\/pre>\n\n\n\nHere is a screenshot showing the results of doing just that (from within an Amazon EMR cluster node).<\/p>\n\n\n\n![\"\"](\"http:\/\/www.wis.co.uk\/blog\/wp-content\/uploads\/2019\/02\/hdfs-df-s3.png\")
<\/figure>\n\n\n\nIt suggests that the available capacity of this s3 bucket is 8.0 Exabytes<\/a>. This is the first time I’ve ever seen an exa SI prefix<\/a> for a disk capacity command. As geeky things go, it’s pretty exciting. <\/p>\n\n\n\nI assume this is just a reporting limit set in the s3 driver, and that the actual capacity of s3 is higher. AWS claim the capacity of s3 is unlimited<\/a> (though each object is limited to 5TB). AWS is constantly<\/a> expanding<\/a>, so it is safe to assume that AWS must be adding capacity to s3 all the time.<\/p>\n\n\n\nThe factor limiting how much data you can store in s3 will be your wallet. The cost of using s3 is charged per GB per month. Prices vary by region,<\/a> and start at 2.1c\/GB per month (e.g. Virginia, Ohio or Ireland). For large-scale data, and for infrequent access, prices drop to around 1c\/GB. Assuming you don’t want to do anything with your massive data-hoard. <\/p>\n\n\n\nUsing “one-zone-IA<\/a>” (1c\/GB\/month), it will cost US$ 86 MILLION a month to store 8 EB of data, plus support, plus tax. If you want to do anything useful with the data, a different storage class might be more appropriate, and you should also expect significiant cost for processing. <\/p>\n\n\n\nJustin – February 2019<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"I remember when I thought a gigabyte was a lot. I bought an external 1GB hard disk in 1995, and filled it up in no time at all. As geeky things go, it was pretty exciting. Hadoop is designed for … Continue reading