Transform for Code Profiling #646
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This tranform extracts the base syntactic concepts from the multi-language source codes and represent these concepts in an unified langauge-agnostic representation that can be further used for multi-lnaguage data profiling. While programming languages expose similar syntactic building blocks to represent programming intent, such as importing packages/libraries, functions, classes, loops, conditionals, comments and others, these concepts are expressed through language-specific grammar, defined by distinct keywords and syntactic form. Signed-off-by: Pankaj Thorat <[email protected]>
pankajskku
force-pushed
the
dev-pankaj
branch
7 times, most recently
from
eea6e72 to
47b9dcd
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
pankajskku
force-pushed
the
dev-pankaj
branch
21 times, most recently
from
cc43bb7 to
6294b2d
Compare
Signed-off-by: Pankaj Thorat <[email protected]>
pankajskku
force-pushed
the
dev-pankaj
branch
3 times, most recently
from
dba3567 to
627b4db
Compare
|
@daw3rd Please let me know your opinion on the updated PR. |
touma-I
requested changes
Oct 12, 2024
There was a problem hiding this comment.
@pankajskku Please slack me when you have a chance. Internal ID: [email protected]
pankajskku
force-pushed
the
dev-pankaj
branch
2 times, most recently
from
46d3e2d to
7e183d0
Compare
Signed-off-by: Pankaj Thorat <[email protected]>
pankajskku
force-pushed
the
dev-pankaj
branch
from
7e183d0 to
4f0bdd4
Compare
pankajskku
changed the title
Signed-off-by: Pankaj Thorat <[email protected]>
pankajskku
force-pushed
the
dev-pankaj
branch
from
143c054 to
39158a5
Compare
touma-I
requested changes
Oct 15, 2024
touma-I
requested changes
Oct 15, 2024
There was a problem hiding this comment.
@pankajskku Please see additional changes. I was pulled how your UT was passing without the proper requirements.txt in the Dockerfile.
Co-authored-by: touma-I <[email protected]>
Co-authored-by: touma-I <[email protected]>
Co-authored-by: touma-I <[email protected]>
Co-authored-by: touma-I <[email protected]>
Signed-off-by: Pankaj Thorat <[email protected]>
pankajskku
force-pushed
the
dev-pankaj
branch
from
11cb4d5 to
efcf6d0
Compare
touma-I
approved these changes
Oct 16, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This tranform extracts the base syntactic concepts from the multi-language source codes and represent these concepts in an unified langauge-agnostic representation that can be further used for multi-lnaguage data profiling. While programming languages expose similar syntactic building blocks to represent programming intent, such as importing packages/libraries, functions, classes, loops, conditionals, comments and others, these concepts are expressed through language-specific grammar, defined by distinct keywords and syntactic form.
Why are these changes needed?
Data profiling, in the context of machine learning, is the process of examining and analyzing data to create
useful statistics. These statistics are used both as an aid for better comprehension of the properties of data as
well as for a variety of downstream data processing tasks such as data valuation (assessing the value of data
relative to the business objectives at hand) and data curation (filtering and prioritizing training data based on
derived thresholds). In the Large Language Model (LLM) setting, training data is typically unstructured in
nature comprising natural language text, images, and code. In this work, we specifically focus on code-LLMs,
where the quality of code training data substantially affects the model accuracy of LLM-based coding tasks
such as code generation and summarization. Therefore, having the capabilities to characterize code data in
terms of programming language concepts aids in both deriving insights related to code training/evaluation
data and in the downstream curation of code training data. In this work, we address the problem of profiling
multi-lingual code datasets by extracting an extensible user-defined set of syntactic concepts
over arbitrary programming languages.
Related issue number (if any).