From 35646268d79fcf6dfd0514104423fdb6e5fdb11d Mon Sep 17 00:00:00 2001 From: chandan Date: Thu, 16 Jul 2026 15:59:26 +0530 Subject: [PATCH] Add XLS/XLSX (Excel) data source support with febrl example and docs Adds a febrl test dataset in .xlsx format, an example config using the com.crealytics.spark.excel format for both input and output, and documentation covering how to add the spark-excel jar and configure Excel read/write. Fixes #19. --- docs/SUMMARY.md | 1 + docs/dataSourcesAndSinks/excel.md | 85 +++++++++++++++++++++++++++ examples/febrl/configExcel.json | 94 ++++++++++++++++++++++++++++++ examples/febrl/test.xlsx | Bin 0 -> 8540 bytes 4 files changed, 180 insertions(+) create mode 100644 docs/dataSourcesAndSinks/excel.md create mode 100644 examples/febrl/configExcel.json create mode 100644 examples/febrl/test.xlsx diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 8584c3ce3..80247c42e 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -65,6 +65,7 @@ * [MongoDB](dataSourcesAndSinks/mongodb.md) * [Neo4j](dataSourcesAndSinks/neo4j.md) * [Parquet](dataSourcesAndSinks/parquet.md) + * [Excel (XLS/XLSX)](dataSourcesAndSinks/excel.md) * [BigQuery](dataSourcesAndSinks/bigquery.md) * [Exasol](dataSourcesAndSinks/exasol.md) * [Working With Python](working-with-python.md) diff --git a/docs/dataSourcesAndSinks/excel.md b/docs/dataSourcesAndSinks/excel.md new file mode 100644 index 000000000..d14d7e89f --- /dev/null +++ b/docs/dataSourcesAndSinks/excel.md @@ -0,0 +1,85 @@ +--- +layout: default +title: Excel (XLS/XLSX) +parent: Data Sources and Sinks +nav_order: 12 +--- + +# Excel (XLS/XLSX) + +Zingg can read and write Microsoft Excel files (`.xls` and `.xlsx`) using the +[spark-excel](https://github.com/crealytics/spark-excel) connector. As Spark does +not support Excel natively, the connector jar has to be added to the classpath at +runtime. + +## Adding the spark-excel jar + +`spark-excel` is not bundled with Zingg. Pick the artifact that matches your Spark +and Scala versions — the coordinate is +`com.crealytics:spark-excel_:_`. +For Spark 3.5.x with Scala 2.12 this is +`com.crealytics:spark-excel_2.12:3.5.1_0.20.4`. + +Add it in one of two ways: + +1. **Through `zingg.conf`** — download the jar (and its transitive dependencies) + and list them under `spark.jars` in your + [runtime properties](../stepbystep/zingg-runtime-properties.md) file: + + ```properties + spark.jars=/path/to/spark-excel_2.12-3.5.1_0.20.4.jar + ``` + +2. **Through `spark.jars.packages`** — let Spark resolve the connector and its + dependencies from Maven automatically: + + ```properties + spark.jars.packages=com.crealytics:spark-excel_2.12:3.5.1_0.20.4 + ``` + +## Reading Excel as a source + +```json +"data" : [{ + "name":"test", + "format":"com.crealytics.spark.excel", + "props": { + "location": "examples/febrl/test.xlsx", + "header": "true", + "dataAddress": "'febrl'!A1" + }, + "schema": "recId string, fname string, lname string, stNo string, add1 string, add2 string, city string, areacode string, state string, dob string, ssn string" + }] +``` + +## Writing Excel as a sink + +```json +"output" : [{ + "name":"output", + "format":"com.crealytics.spark.excel", + "props": { + "location": "/tmp/zinggOutput.xlsx", + "header": "true", + "dataAddress": "'output'!A1" + } + }] +``` + +## Useful options + +The values in `props` are passed straight through to spark-excel. Commonly used ones: + +| Option | Description | +| --- | --- | +| `header` | `true` if the first row holds column names. | +| `dataAddress` | Sheet/cell range to read or write, e.g. `'febrl'!A1`. Defaults to the first sheet. | +| `inferSchema` | Infer column types when no `schema` is supplied. Prefer supplying an explicit `schema`. | +| `usePlainNumberFormat` | Read numeric cells without scientific/locale formatting. | + +Both `.xls` and `.xlsx` are read and written with the same +`com.crealytics.spark.excel` format string; the file extension in `location` +determines the workbook format on write. + +A ready-to-run example config is at +[`examples/febrl/configExcel.json`](https://github.com/zinggAI/zingg/blob/main/examples/febrl/configExcel.json). diff --git a/examples/febrl/configExcel.json b/examples/febrl/configExcel.json new file mode 100644 index 000000000..57060fb90 --- /dev/null +++ b/examples/febrl/configExcel.json @@ -0,0 +1,94 @@ +{ + "fieldDefinition":[ + { + "fieldName" : "recId", + "matchType" : "dont_use", + "fields" : "recId", + "dataType": "string" + }, + { + "fieldName" : "fname", + "matchType" : "fuzzy", + "fields" : "fname", + "dataType": "string" + }, + { + "fieldName" : "lname", + "matchType" : "fuzzy", + "fields" : "lname", + "dataType": "string" + }, + { + "fieldName" : "stNo", + "matchType": "fuzzy", + "fields" : "stNo", + "dataType": "string" + }, + { + "fieldName" : "add1", + "matchType": "fuzzy", + "fields" : "add1", + "dataType": "string" + }, + { + "fieldName" : "add2", + "matchType": "fuzzy", + "fields" : "add2", + "dataType": "string" + }, + { + "fieldName" : "city", + "matchType": "fuzzy", + "fields" : "city", + "dataType": "string" + }, + { + "fieldName" : "areacode", + "matchType": "fuzzy", + "fields" : "areacode", + "dataType": "string" + }, + { + "fieldName" : "state", + "matchType": "fuzzy", + "fields" : "state", + "dataType": "string" + }, + { + "fieldName" : "dob", + "matchType": "fuzzy", + "fields" : "dob", + "dataType": "string" + }, + { + "fieldName" : "ssn", + "matchType": "fuzzy", + "fields" : "ssn", + "dataType": "string" + } + ], + "output" : [{ + "name":"output", + "format":"com.crealytics.spark.excel", + "props": { + "location": "/tmp/zinggOutput.xlsx", + "header": "true", + "dataAddress": "'output'!A1" + } + }], + "data" : [{ + "name":"test", + "format":"com.crealytics.spark.excel", + "props": { + "location": "examples/febrl/test.xlsx", + "header": "true", + "dataAddress": "'febrl'!A1" + }, + "schema": "recId string, fname string, lname string, stNo string, add1 string, add2 string, city string, areacode string, state string, dob string, ssn string" + } + ], + "labelDataSampleSize" : 0.01, + "numPartitions":4, + "modelId": 100, + "zinggDir": "models" +} diff --git a/examples/febrl/test.xlsx b/examples/febrl/test.xlsx new file mode 100644 index 0000000000000000000000000000000000000000..fbb72325e8d20d37e4793ed3b61915548fd11c86 GIT binary patch literal 8540 zcmaJ`1zeQf(g&#}-33INrI!vD1<8d4mTmz-loFOMDFLOVyOD0C1nEXF77&z9=|+)I zLV@pDz4yL;@AuvJ@Oz%;95^#)=6_~}GiS8b@bIZ|Kp+s#fcIBZoIe^B&^J+VcJi=x z^0=$-<6`Z8N6_2Rp6I|}XUe+?-s^|j23G%QfEoBM{}tr}b#ZY#)B`SrZ<>*f8*YrEX~#Gl`6>)|c)R5LXODYygayZa*; z3kF=i=ieq@$gCej@t-qiA(`2{<)`Db20YZW!*vC$iIzO!EmXY6WHJ-Ywl6kUh;KL7 zHv7)bUEO?wp{>*r)#I=Eeu}F5M7odfxp2>!?GrfAV2D9?s$Kym>iVR|)m%bvj)9zu zj{G8$W_ld<0a3}A;sSe4@4-X`(YyX)NFt4=Q&l-A=$*T~PCUG`_lGde70N7YqwuQ| z%nzbO0!JtDtZXE{iv|V^MVO83f*JS%z*Xt&&YvTAl5z0d-V=Z5%Z8sP|1d=L4}nuB z4umLr&V_#9B1QO@6(~gfb~kTms}X?`_+f35xHvdW_&7M){}p>u0Q8ie&Vlx)jsy_&bQGvPVw8wBX=?7pQ+1TmTt|C%RD%Peds+C}{Dla{Q#MSc zT~9ElFz~@hP=W2#$s~ifgiO^JxhvfpE=wpvP5pbFGcqF~x`urVdO9hPs6YhAoc*00 zIb5}#2#e1iS$L`f?$uP&kcY!h8`{TCxbX%vwid`%f=z;NT6|F<8%R1|km?YP(|A4p zhUcTReDii&F;RnaU;)3?UG>Lio>P9U;tg6E+gv4w^P+H4MP;WHyGz!4-l1ge#%w1zh1SBd77_2Mm>=Ba-kkpxhj;$hWU^z)ib0EBG> zp8vktkU!P2bau1G>f;zcquMJ>t@wy@rCnp%V(fz_*ji4T@rCG;2`bjHra2xdd2m$n zpdzn>b84Gk)3aWp4C4RzAo!!$pl&3@)_f(em0q@fX5|t)OH-;JoZik4KkkMGCqLDj z?m;?IL9zv%L))Q{N>`R3>k#jc!AmkRA+d|)u90Eh51499+x93>nrIbjd_kWDD;h3O zv85#4NCWizTob4WDjal)NeVL@{qqHG*8T+rKSq`L=L_8tc(GoMi)&%D)cm}(U_Wct7;2`lai8 z1O6hSx`$2K&EJ~Y9@#m`;9^nq?^sX(z)BuX!4npp>=5E$jdLC|QCmVMx zkl7|`iOb>?b3;7TpKMAyQb!Y3%{-7^1NU!TBg*5Lw#a0=!(rXHP3T_LJSpedBFZ0c za8D>H6}gY=?TwS++<7L3Ss94Aa@&8(sxmStl3OQr9*~yJ}7y1NRIcT1a2i;%4#Ru<4lM z%(i;_%_>m&eM&yD+I0B~PWJAv7?oFAy(zBlD~WfLUP^)QW~A5PDMwZG0Y=wFl-yo% zcW}Cm5Vify+w9ds1BnszA>G=Dn(EWB7<#G9?>m9P=uXxnBS~{mUVw#NgUEg8rP3G% zsv8XVOQvWW%ncS)6X>#q~!T-rxLjI!u@`@PiCTRG&+Cr<-&FuNUCGvD#mdMhf$eqe+ng-*pnVj z`k)YMKYXcGL9a+760cTZzeKJ_>nkOC!II^=1v{y3gk)z@P0OxSo2y$}o>#7C+Y_O$ zDPH3&m1fs$78V2GmzExrsb7-mRb&(lFv@9A8j${plkryP$n)qJ|LLB)(*R%Gd?aWP zqRxZRu&0>0Z&-?~Z36KqX5H|*%~x5>HQkhE5eFe0E*0G4_hnq;Y|CMNkYZ>}vdHp6 zPNnfN-Xi->)7mna6URIIx^0|^Me^Vvn#vmQ#{SB^Ly5$2LDS}bfz_RiWLQw2;}c3A zj!8!3vM*mN;et>Dsy{&P&d2<D};ZZH{&QfGJK@eYINK+ zgVujB4>J%^ModFY?nhwtMK~eNgpJ@)O4;>5THC8UFPWXoOP+1Vf1iD0JJd;=5=!xy zI{~RhninC?xic`Pj)Wu^(xxMa8~TRp_i)n-Gy++`!akYB1}Hp-c1n9?YJx_M*I%A( z%kiJaw+hssbnEsQmy{6YSU4dSO~MPf7D!>JQi=>H_stP=Pbm2lyt{`rp93SpQ#e@` z$Q?iHo)yxP(2&?QaWUO$)V>1c4d$hY>|%_-wDcV?DsJFyzF|-Sk05!+r$M3RYhaS4 z0rIwB+q|-V8fTdOn4Jy6KCNC*dm1VRnjNA9I@HqD^dAFB~^yd~>;0MB*+t zh!lCYscKfOA4kO5b!`>1zQPaar<$=}Pc%ypph1@I*3)^sjU^5yNkDOX&XUF+AcwAWEKNl70H60u^fg9DCIGHepk$R^?_35g^N(t^l5!bHLj_CSG5Lo zga-0ixgQ6nEPxc)#=pTlrce>zG3bEt3~MS#l;D=|W@DdT%8QZljU^p+sI&^YW+ACx z*|DJE9=_hDE91K5F)1=wc zJBF>Y`CiUojd>aD+sQvHHxBcAU!7Yy+*4pYlYFmF&L%%xRZjU_L;IjRf_U}#w&sqg z#YWf6t)N%oyto3pR0TP1(|+GPwn^g#h)(7O`GdDI2uT(k!;{$(2mPx9D`h$i-_S-f zJXcMhG!S`7qlYLE4pL$BQFUFeF)0o$$ghGlS~;4Oa@xJ#-4Z48VcMS`p0_F6IJ+%L z(!);Y8)zQhgP^@kVVq>(7)3Hz+73}Xe3u74fTb-o_GH#Yy4HJfeHC$e;g>||$7IdR z#@%Ghd_1OCA6BkN+VS4NR5l0GZ!i}}nJ~XL8W2()KEztM_AX)Du|r@Q9HSp_;S|Ap zPB^u?h4zs0&)u-RC(D{}CU4~j;SE1KI7H_Ay61-6yy!9Edfr;*?~<@+6k&%G*W^In z(9DfmQ!q^n)`H$zjc0Vy)VdPqCX!fMPqdmAR+vU1BGV4DdlwS!o-4w3<0hF5Y&wPY z{7B4M0qwaZjQ7Zot6*rxUePJZqfOQfe+}~d9@EJZ{yFHmR3Zo7SdR5EqwLVgbw`Eo z5w=hY_56MD5UM6&{hq|QyAi_zVV>tGhqKYbl`<)mmxAV8~ksqhH=pc2L4 zVyRzGoien+Lwl9U1;J!GLFaTT70)sAv*wfq6e}7dGSHg^-ToQT(|fAU;VG?9!@JQu z=R#!Nhd-E+MKGD<8#aijETA}!&3j8`Jj0(R2ZpQ})Vz=E;(0@qH!7Jxs$?JTi4=I) z!=D$PN+_HN@~IaHXmEMH@86XxF8ee0F1g)n(JuwO$?nLMnj5FtWqhr>hU|s(d~b=p zIV1#_mYi2UJyRX%{V3oB@~mDDltR?jI@L+ z(`&CwCWoabS#-v2zIr-!v#P5^Ei7X)O%jCn4OexuwLnp9m56;|rb{v5Mvl@H6>>Cq zclFd}z-fJuqvLbHATvc*nPl+SGU#0+OZJ*i^xf>~?G&y|+%FkTHbUml@31Dc7_P{0 zCnb6vxGWswmsGfLrqzRUbAF&5v(p|>k-UaA7b2^lT@TZB*{DN$Xi3MvJcB99zTu(g zD@h`dcsEdI>Wf>u6E|8CJbtyZl!AK}{%td#&2jHR5Me;sUVA2Y#j~-)x94M*6?OCr z_auuRUH-UodIM~bA1-Kls?1V;kIQy!Xh;U~y1d@$=n?kZW&MuY@;WgNjg$AJw0R8Hi*yYw2Cwrn(!{^}GIBL~q)0DJ z)9c9n;_c$K`Izx|(&Id)agX-rFkFYM;mNVXv!+M0!`a>oteS)r(sCS_$N44x(LL8J z^EbQuUbx}iyNS+j6fGI{;wNJ+BAK!xpV1$qda28Pg>lH!*%BHfX<30*%vh-Gd0Jbd z2$u3oh=QwN3X2=w(5*)(5wMqKI=y@{EWhcSpp@H|<1=h`?TSxQsLycmke!E2RK=`l zSiJgldM@krI#AGMQKhdbrvRLI3;r<`y8Ih`FK0J<3ukA0?148X zwqLVH7_4~Y7b^D9jtCqC)4X9&{Kbg+JC)6_(KmMbr>R44Rj%B_^tH6u#5%t`+t)M= zPl`b()ayWD>60@q!(Rht=R3O^+U=_nhVDe$=zNWbM@wkcCu5>lL>!lp`EUiRxMrjb z1}|2NH^d$3QIlSi{`R9U@L+X8vS1ZD4HRw9Mpzk(Q3{lM2463(O zdL9cx-Zf-Ay{tBINIIyKhxsnAyaO-GNP5&|eQ+B2I43gxrYE$ip`B3XFb}3NUzeo{>*TfXp_jgve_KeFBNgi@Ue|>nl&h$}C@J*dH8A3b_gW!UDh1&`?3okCb zeFyy@;#hs zQ}IschF||<2|dTGz&2O%GU>J0UYv&mE1rpJ^o9hcxAY1wY%OQLr+z9tW8`_q)SAgl|1_KM>xxx0$-)){i9FB znn7F(@4OSwPV7Zr3VILnEVkw`b{c$Hwww5I%kboQ{X)iX>PLf-^qdfjmeTvyJzM9! zV{;#`i{nkJ!`*qoivaP{lSAjX^J(StxiUgD+wJ5R#~z}mZDxdI1#%$*KIaz+@T!vW_0hyfW z0CYf0wx;rr(zmh;7qdbgvhBx5pWJ75)B5LgFPb_e=^R^{ayl;_>{;)bw@?0TH#@8G z4)`Wx7I0E1d13l_PC#bx;<)yFV*kh3()q~xx3%o=Pa97!xQ5oZPd4}LPW+wgPjWRb z%Sv4=r`;dT*P{@BhXh*CGBO5-@4 z=#B0@6mwNvS`c&9So$jFs<5;n=E`SSVLEyi6I2=Ixt+giza)A&sP};k6^GzqGt{+M z2tPv2p9f{SodI>Nsm6~e_ZL8!eqms9^n2n=fSyG)iVg3Hxi%9amfUiQDbs$K#y8Dg z4eDL(pym)fQU?Zx*a6q@>(vejhs;1@P@R)it(fSd{gT^d2A zKE@;!o~*>7N)O?P0qKeiJI5UQY4~jrB9;`qxKM4c-Pw}7qX6zEZPXl34%Oq2aqbXP zvgFzOD5H@Dy`{^y;-4{OIK`-+WWwd~e{%&`(YH`@&}Tw63-kF9@@MtX{PQQMPYCDY z)+_8pChSBG61%s9dbzg%c1>ASfe^%SI6QfNCurqHD-<09dK2z(<83(f3VWCdd)SVt zX(%Ed&W4JGunDuF=vhT$VAnR)VgVu$gf`(z2?W8b6x*N_drag0UG0|$fRb1O4=*l% z751PaEFHF^9}Qnlq>bI=SnqtZ;?bX;rZ_E_;Sf+xo22k2{im78{Y=jg36$h3#u6s;*x}ffjpaIa*L%uKU9rMqlAIaN z>CVirVufF@lkI@;Bwqo)#dhN*f|$%R>)6$wdWLNBda7E?3m-{k7BQKh*0B%3KeuJ4 z-Zf&JoMLTn6mNP-94yH|H)}D^2N|P^rHd6) z5PPuQa#`{v`upX;Jf_fC-9)+f;sSJ|MV!IYbynJGsbrZ&T*0S@9Agv%SNK1`rRhAh z_w-5d$=xTpwiklsP6%8Y@Z76nn2L)V@$THtt7sjfw$n{SII6Ps@&prX0o|m?BHe4Y z>EqXpshRN$ikP+t$7wl@@v#h}UkOEFpfX3jBu8AtCfZivr-KJpq=KTt7K^eZ}5u#AH(>!fW7V16& zrcgJ5+fa@gdYH^!BMFm%R0^8{AL*2_AOm#}ckqG=)K?UgQ_-9y2TP^`Ia1_!!misw zy+zB`HebKQYbeQ`*C5dtNkV+J(xU6ae%^N{A-og2GK0D053K?hFnO7!y56>&EQ8bg zz>4q4`%d?p5rD~&Jtz%A6xtV(;Z|4QF}j=Y4BA#8p1(&7&lywx_-{Jyrfhc(@Z+Eddp z2C@Rc1~5?XTA9_OBqgjt0E6j#=KO!{Y}|$wsa#bC1&R5!NW^3;+mv{!7{6|b)-*jF z*jPfu2*oOh9O#U}0Ri6bG`&~M>W10@B34NFNz;W?a|a7Y34`uxjk^L|8L?bjMh&sT zau1!Xw`}HV3sTY}wP-elrRngsoSn7P(P_e2WIc+2P3Gh!AN6w$H-b#^Mnz*xtuXaxG1QAEC?11?Yn6P5#w16k0|=k zEG$inuS3tX{eqrHIa%q|6T_(FZ7mY>k#Yo&YCd(bcI2>^A(a6MC6&Z6POAWU0W8WR zy?~z*XHmqUH}sG^dRnRT760X5`b?5-=!Yx1_cWXT?G*XV71KxlENCLc|w+L0SS!qOe;DJ z%4&Dfy1bNBm^k+nUu}>w0bNK>5ZNnN#D81{UrRFU=w6^(@!_p}AjZU}qkOaMOM`*f zlJNB#HYBhuZey@+yEOv1ZYh&y;D>P_97UFPjI#S;V%_7`^0Wv0uJpktpcPbhML&Hh z%lbn%GUS^pmGWnAv4g)nq=z+*!SttYjik9Dg-ZF?YB!+Z+?UiSDPpA7cxl@%+TDjN z`Il3m!#)E#=8!%20qilS*)^^@$8LHt z9z+gT(I7M!C;p)uwUNQ#)r`DGi`1auu~{WqtuSSs{Z95P{N|YschH3ARET-kj@#YU zDyIE7(;X@iBOIPX_R6KUka`j8oU!XIPq1U5I?sZu=dDrVnz>q;jkSZ1{bc z!xVWR_)|o>qEVhyFDk!}N6G)0(xF>`g&;W@p-z$_KqM!AU)AK7p{Q?$*SJuT9<-X5tNWh+T!SQfSlb<>JLG;{e?Y0K1AfU)7^xADmu zBs?62QzGf~sSKV2>|@G7!%X1x=mp^ZEt4zHXm~%!Q*2f0Wi|G_Ii`27Jzr*i?MkZQ6icCaN zTl)hMXMYp;2h83F4NrV~^WRg20*2)&S4@|&oS%tlCnF1B+f=}+v!}melxj{~9xMt7 z8O3F1S7rIWKR8G4#hq-8oGp)h`myTU?l;-?Wv=_p(c4wGb|dMF`sVBavw_`)M}EeB zrWbii56fjARqwp*oanE9)c*ZseXyYu<9RBJ&6iT!w%%`%;o!V+#NKby_uf|RBo5O!DivCXeXL66d z*Z8Xy14Cf(KU4X?6aSg~V)OsMS_^<8K>S~B1OCqEPou!5%73*90MCC3@qZF$Z8ZWy R>`Y?dM;6#7A{OlEe*kLq;feqN literal 0 HcmV?d00001