Cov ntaub ntawv hauv zos ua tiav li cas hauv Hadoop?
Cov ntaub ntawv hauv zos ua tiav li cas hauv Hadoop?

Video: Cov ntaub ntawv hauv zos ua tiav li cas hauv Hadoop?

Video: Cov ntaub ntawv hauv zos ua tiav li cas hauv Hadoop?
Video: Data Science with Python! Analyzing File Types from Avro to Stata 2024, Kaum ib hlis
Anonim

Cov ntaub ntawv localization hauv Hadoop . Siv cov qauv Wordcount piv txwv, qhov twg cov lus feem ntau tau rov ua dua rau 5 Lacs lossis ntau dua. Nyob rau hauv tas li ntawd tom qab lub Mapper theem, txhua mapper tso zis yuav muaj cov lus nyob rau hauv qhov ntau ntawm 5 Lacs. Cov txheej txheem ua tiav ntawm kev khaws cia Mapper tso tawm rau LFS yog hu ua Data Localization.

Khaws qhov no hauv kev saib, dab tsi yog cov ntaub ntawv hauv zos hauv Hadoop?

Lub tswvyim ntawm Cov ntaub ntawv thaj chaw hauv Hadoop Data thaj chaw hauv MapRub hais txog kev muaj peev xwm txav tau qhov kev xam ze rau qhov tseeb cov ntaub ntawv nyob ntawm node, es tsis txhob txav loj cov ntaub ntawv mus xam. Qhov no minimizes network congestion thiab tsub kom tag nrho cov throughput ntawm lub system.

Tsis tas li ntawd, cov ntaub ntawv loj yuav khaws cia li cas? Cov neeg feem coob tau koom nrog HDFS, lossis Hadoop Distributed File System, nrog Hadoop cov ntaub ntawv cov tsev khaws khoom. HDFS khaws cov ntaub ntawv hauv pawg uas yog tsim los ntawm cov blocks me. Cov blocks no yog khaws cia nyob rau hauv lub cev khaws cia units, xws li internal disk drives.

Ib yam li ntawd, cov ntaub ntawv khaws cia hauv Hadoop li cas?

On ib Hadoop pawg, cov cov ntaub ntawv nyob rau hauv HDFS thiab MapReduce system nyob ntawm txhua lub tshuab hauv pawg. Cov ntaub ntawv yog khaws cia hauv cov ntaub ntawv blocks ntawm DataNodes. HDFS replicates cov cov ntaub ntawv blocks, feem ntau yog 128MB loj, thiab faib lawv kom lawv rov ua dua nyob rau hauv ntau qhov ntawm cov pawg.

Cov ntaub ntawv khaws cia hauv HDFS li cas?

HDFS exposes ib ntaub ntawv system namespace thiab tso cai rau cov neeg siv cov ntaub ntawv ua khaws cia hauv cov ntaub ntawv . Sab hauv, a ntaub ntawv muab faib ua ib los yog ntau lub blocks thiab cov blocks no yog khaws cia hauv ib txheej ntawm DataNodes. Lub NameNode ua haujlwm ntaub ntawv system namespace kev khiav hauj lwm xws li qhib, kaw, thiab renaming cov ntaub ntawv thiab directory.

Pom zoo: