How is metadata managed in Hadoop?

Metadata in Hadoop is typically managed by the Hadoop Distributed File System (HDFS) and the ResourceManager of Hadoop YARN.

1. Metadata Management in HDFS: HDFS maintains metadata about files and directories, including filenames, file sizes, creation time, and access permissions. This metadata is stored in the NameNode, which regularly saves the metadata to the EditLog and FsImage files on disk to prevent data loss. Users can manage metadata in HDFS by using Hadoop’s command-line tools or API, such as creating, deleting, moving files or directories.
2. YARN Metadata Management: YARN is responsible for managing resources in the cluster and maintaining metadata information about jobs and tasks. The YARN ResourceManager tracks resource usage, job status, and task status in the cluster, and allocates and schedules resources as needed. Users can manage job and task metadata using YARN’s command line tools or API, such as submitting jobs, checking job status, and terminating jobs.

Overall, the metadata management in Hadoop is performed collaboratively by HDFS and YARN, allowing users to manage and operate metadata information in the cluster using the respective tools and APIs.