Modin running slow than pandas

[pavan praneeth]

Hi first when I ran my code it was fast and later when I ran it second time it became dead slow. Anything reto fix from my side. Please reply fast.

Thanks 

[Yaroslav Igoshev]

Hi,

Can you give the code snippet you are using, please?

понедельник, 22 марта 2021 г. в 13:26:44 UTC+3, pavan praneeth:

[Yaroslav Igoshev]

Discussion was moved to GitHub issue (link).

понедельник, 22 марта 2021 г. в 13:30:09 UTC+3, Yaroslav Igoshev:

[Vikas Garud]

I have a dataframe having >100k records, it has 10-15 columns which needs to be coalesced, I used following code:
for COL in [cols_to_be_coalesced]:
df['COL'] = np.where(df['COL']!='', df['COL'], df['COL_X'])

Here if df is pandas dataframe, it executes it in ~2 minutes but of df is modin dataframe, it takes ~30 minutes.
Can you please suggest equivalent faster alternative of np.where for modin dataframe ?

[Devin Petersohn]

Hi Vikas,

Thanks for the question. It's a good point, we don't currently support the numpy functionalities, but there might be a way we can with NEP18. If you are comfortable, it would be nice to add an issue to the GitHub repo to support NEP18 in Modin.

A small rewrite to enable the parallelism of the low level Modin implementation:

def apply_func(df):

    for COL in [cols_to_be_coalesced]:

        df['COL'] = np.where(df['COL']!='', df['COL'], df['COL_X'])

df.apply(lambda s: apply_func(s.to_frame(), axis=1)

Let me know how it goes!

Devin

This communication is confidential and subject to and governed by Saama’s Electronic Communications Disclaimer.

 

--
You received this message because you are subscribed to the Google Groups "modin-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modin-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modin-dev/f413bb96-adcd-41ae-99ea-a1f15dd67682n%40googlegroups.com.