how to save weights

I am using PSPNet for multiclass segmentation. Training is fine and metrics are good. In the end I am saving model using model.save_weights(path)
But after loading the model some weights are nan (to be more precise, I have nans in level1_bn layer) Can anybody suggest solution for this problem?

Is predictions differs for actual model and loaded?

I am using PSPNet for multiclass segmentation. Training is fine and metrics are good. In the end I am saving model using model.save_weights(path)
But after loading the model some weights are nan (to be more precise, I have nans in level1_bn layer) Can anybody suggest solution for this problem?

Can you post your result, cause I'm not getting a good result on cityscapes?
I train the network for 30 epochs with frozen encoder, then unfreeze the encoder and resume training, but no problem in saving weights.
But the saved model has no optimizer and loss
Segmentation_models,Classification_modesl,Efficient_Net : all the latest version
my training code
from segmentation_models import PSPNet
from segmentation_models.metrics import IOUScore
from segmentation_models.losses import JaccardLoss
from keras.callbacks import ModelCheckpoint
from keras.callbacks import TensorBoard
from keras.callbacks import LearningRateScheduler
from keras.utils import multi_gpu_model
import os
import argparse
from CustomGenerator_il import CustomGenerator
import tensorflow as tf
from utils import PolyDecay
from keras import backend as K
from keras.optimizers import SGD
from keras.callbacks import CSVLogger
from segmentation_models.utils import set_trainable

parser = argparse.ArgumentParser()
parser.add_argument('--batch_size', type=int, default=16, help='input batch size')
parser.add_argument('--image_width', type=int, default=384, help='the input image width')
parser.add_argument('--image_height', type=int, default=384, help='the input image height')
parser.add_argument('--lr', type=float, default=0.01, help='learning rate of the optimizer')
parser.add_argument('--decay', type=float, default=0.9, help='learning rate decay (per epoch)')
parser.add_argument('--n_epochs', type=int, default=100, help='number of epochs to train for')
parser.add_argument('--n_gpu', type=int, default=1, help='number of GPUs to use')
parser.add_argument('--n_cpu', type=int, default=12, help='number of CPU threads to use during data generation')
parser.add_argument('--checkpoint', type=str, default=None, help='path to model checkpoint to resume training')
parser.add_argument('--epoch', type=int, default=0, help='epoch to start training')
opt = parser.parse_args()

print(opt)
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)
K.set_session(sess)

checkpoint = ModelCheckpoint(filepath='output_il/epoch_{epoch:03d}--val_iou_score_{val_iou_score:.3f}--val_loss_{val_loss:.3f}.h5',
                             monitor='val_loss',
                             mode='max')
tensorboard = TensorBoard(batch_size=opt.batch_size)
lr_decay = LearningRateScheduler(PolyDecay(opt.lr, opt.decay, opt.n_epochs).scheduler)

train_generator = CustomGenerator(mode='train',
                                  batch_size=opt.batch_size,
                                  crop_shape=(opt.image_height, opt.image_width),
                                  resize_shape=None)
val_generator = CustomGenerator(mode='val',
                                batch_size=opt.batch_size,
                                crop_shape=None,
                                resize_shape=(opt.image_height,opt.image_width))

optim = SGD(lr=opt.lr, momentum=0.9)
loss=JaccardLoss(class_weights=None,class_indexes=None,per_image=False)
iou=IOUScore(class_weights=None,class_indexes=None,per_image=False)
csv_logger = CSVLogger("model_history_log.csv", append=True)

# Model
if not opt.checkpoint:  #From Start:
    net = PSPNet(backbone_name='resnet50',
                       input_shape=(opt.image_height, opt.image_width, 3),
                       classes=20,
                       activation='softmax',
                       weights=None,
                       encoder_weights='imagenet',
                       encoder_freeze=True,
                       downsample_factor=4,
                       psp_conv_filters=512,
                       psp_pooling_type='avg',
                       psp_use_batchnorm=True,
                       psp_dropout=None)

elif opt.checkpoint:    #Resume From Checkpoint
    net = PSPNet(backbone_name='resnet50',
                 input_shape=(opt.image_height, opt.image_width, 3),
                 classes=20,
                 activation='softmax',
                 weights=opt.checkpoint,
                 encoder_weights='imagenet',
                 encoder_freeze=True,
                 downsample_factor=4,
                 psp_conv_filters=512,
                 psp_pooling_type='avg',
                 psp_use_batchnorm=True,
                 psp_dropout=None)
    # OR
    # net = load_model(filepath=opt.checkpoint,
    #                  compile=False,
    #                  custom_objects={'jaccard_loss': loss})
if opt.n_gpu > 1:
    net = multi_gpu_model(net, opt.n_gpu)

# Training

if not opt.checkpoint:  #From Start
    net.compile(optim, loss, metrics=[iou])

elif opt.checkpoint:    #Resume From Checkpoint
    set_trainable(net, False)
    net.compile(optim, loss, metrics=[iou])

net.fit_generator(train_generator, steps_per_epoch=None, epochs=opt.n_epochs, callbacks=[checkpoint, tensorboard, lr_decay,csv_logger],
                    validation_data=val_generator, validation_steps=None, workers=10,
                    use_multiprocessing=True, shuffle=True, max_queue_size=10, initial_epoch=opt.epoch)

@ali-masoudi Training PSPNet also required an auxiliary output, please refer to original paper.

hello, I may have a relevant question,
does loading weights by setting weights=(checkpoint model) in sm.Unet(......,weights=ckpt,encoder_weights='image_net', .. ) overwrite the image net encoder weights?


Fix vfwmul.vv

I was just confused as to why for XLEN=32 there was a testcase for vfwmul.vv generated.
Then I spotted this issue.

Thank you.


Opencl indexing

Submission Checklist

[ ] Run unit tests: ./runTests.py src/test/unit
[ ] Run cpplint: make cpplint
[ ] Declare copyright holder and open-source license: see below

Summary
Adds indexing for kernel generator (OpenCL) types. This was made to match indexing on CPU as closely as possible, although there still are significant differences.
matrix_cl and kernel generator expressions do not store compile time information about their shape. So we can not make different overloads for matrix, vector, row_vector and array types. The way this is made now everything is assumed to be 2d matrix. This will work for matrix, vector and 1-dimensional array types. For row_vectors we will need some help from the stanc compiler. row_vector[index] will need to be replaced by row_vector[index_omni, index]. Similarly indexing vector and row_vector types with index_uni will need to be changed. vector[index_uni(i)] -> vector[index_uni(i), index_uni(1)], row_vector[index_uni(i)] -> row_vector[index_uni(1), index_uni(i)].
For index_multi we need to have the array of indices on the OpenCL device. To keep things simple I decided to achieve that by replacing index_multi with matrix_cl<int> containing indices in a single column. This way the OpenCL version of the index_multi can be made by to_matrix_cl(index_multi.ns_).
This also fixes a bug in rvalue_index_size and rvalue_at when called with index_min_max with reverse order (.max_ being smaller than .min_). Some tests that were passing on wrong behavior were also fixed.
Intended Effect
Indexing OpenCL types is possible.
How to Verify
Run the new tests.
Side Effects
Documentation
In doxygen.
Copyright and Licensing
Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company): Tadej Ciglarič
By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

For row_vectors we will need some help from the stanc compiler. row_vector[index] will need to be replaced by row_vector[index_omni, index]. Similarly indexing vector and row_vector types with index_uni will need to be changed. vector[index_uni(i)] -> vector[index_uni(i), index_uni(1)], row_vector[index_uni(i)] -> row_vector[index_uni(1), index_uni(i)].

It will take a bit of fancy footwork but I think I can do this in the compiler. I started messing around with adding compile time size information in the compiler. I wonder if it would be easier to implement this in math if instead of allowing for general compile time sized matrix_cl types we just support knowing if we are looking at a matrix (-1, -1), vector (-1, 1), or row vector (1, -1). Then I think at least the scope of things needed to change shrinks a good bit (or at least the logic does).
https://github.com/stan-dev/math/tree/feature/compile-time-sizes-for-matrix_cl

I wonder if it would be easier to implement this in math if instead of allowing for general compile time sized matrix_cl types we just support knowing if we are looking at a matrix (-1, -1), vector (-1, 1), or row vector (1, -1).

I think that would be quite a significant refactor, maybe even also affecting stan and cmdstan repos. Is it that much work to handle sizes in the compiler?

I wonder if it would be easier to implement this in math if instead of allowing for general compile time sized matrix_cl types we just support knowing if we are looking at a matrix (-1, -1), vector (-1, 1), or row vector (1, -1).

I think that would be quite a significant refactor, maybe even also affecting stan and cmdstan repos. Is it that much work to handle sizes in the compiler?

Yeah it would be, let's continue with the scheme you proposed. If I have trouble doing this in the compiler we can think about that

Oh and also the tests for the Stan repo that turn on the OpenCL tests

Ack sorry clicked wrong button

Alright I think this all makes sense! At this point I think all we need is the OpenCL tests on Jenkins for this stuff and it should be good to go. I'm fine with doing that either in a separate PR or this PR

Hey @SteveBronder today I had an insane day, I'll try to do the Jenkins part tomorrow my morning.

Word no worries!

Do we just merge this and work on the Jenkinsfile separately? I think that makes most sense wrt to reviewing the Jenkinsfile changes and all. I can work with @serban-nicusor-toptal on that.

I would prefer if the tests passes on Jenkins before this is merged. Also reviewing Steves PR with varmat assign bugfixes I got an idea, how I might be able to shorten the code here a bit.

Ok cool, @serban-nicusor-toptal lets make a PR towards this branch then if you dont mind.

Nice that's a big cut! And yeah I agree on Jenkins tests. I'll give your changes another look today

Good to go?

I have another request. Please add
#ifdef STAN_OPENCL
#include <stan/model/indexing/assign_cl.hpp>
#include <stan/model/indexing/rvalue_cl.hpp>
#endif

to https://github.com/stan-dev/stan/blob/develop/src/stan/model/indexing.hpp
This way its automatically included in the model.


v6.0.1: chore: update dependencies

What does this PR do?
Updates all the project dependencies to their latest versions.
How should it be tested manually?
yarn test
# or
npm test

Pull Request Test Coverage Report for Build 106

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 103:
0.0%

Covered Lines:
46

Relevant Lines:
46

💛  - Coveralls


Add 'self.ref' to repr()

I think it would be useful to add the 'ref' slot to the __repr__ method, don't you think?
Related to this question:
http://stackoverflow.com/questions/36497902/referencing-with-geoindex

@hvpareja sorry i missed that PR

No problem @gusdan . Thanks for the response. ;)


Join invite not available for mentors and unable to accept join requests from mentors.

RTP Dojo Champion, and Founder, (https://zen.coderdojo.com/dashboard/dojo/us/rtp/rtp-1) is unable to invite mentors to join nor can they accept join requests.
See two errors;

Need to confirm that the user was definitely logged in as the correct account when trying to accept the request.

Logged in as user and errors still occurring.
User has been logged in as correct account when trying to accept account and only users available to invite are Parent/Guardian, Ninja Over 13 and Ninja Under 13 as in screenshot above.

The issue was that she didn't have the "mentor" user type on her account. Non mentors cannot invite mentors at the moment.
I think this is a little confusing though as she was also a champion. So I'll update this issue to turn it into an improvement: we need to make it so that champions are allowed to invite mentors also.

We need to also allow Champions to add 'Champions'.

Deployed !


Clients cannot connect to my server

Hi,
I correctly installed Electrumx (clearnet) on Raspberry 4 8go as well as Btc core v22 with onlynet=ipv6(i don't want onion).
I would like to solve these problems please:
WARNING:Env: reduction of the maximum number of sessions from 2500 to 674 because your limit of open files is 1024
Do you know where the file is to modify this directly?
Another thing, I don't get any incoming connection, I use electrumx_rpc getinfo:
 "tx hashes cache": "0 lookups 0 hits 0 entries"
Peers works, I have 92 good peers, my personal ip is also present.
Thank you ✌

WARNING:Env: reduction of the maximum number of sessions from 2500 to 674 because your limit of open files is 1024
Do you know where the file is to modify this directly?

Depends on how you are launching electrumx. e.g.:
https://github.com/spesmilo/electrumx/blob/fb037fbd23b8ce418cd67d68bbf8d32e69ecef62/contrib/systemd/electrumx.service#L10

Another thing, I don't get any incoming connection

Can you connect to your own server?

WARNING:Env: reduction of the maximum number of sessions from 2500 to 674 because your limit of open files is 1024

Do you know where the file is to modify this directly?
Depends on how you are launching electrumx. e.g.:
https://github.com/spesmilo/electrumx/blob/fb037fbd23b8ce418cd67d68bbf8d32e69ecef62/contrib/systemd/electrumx.service#L10

****hi, thanks for your help.
I looked and my file is ok: 8192.
It maybe not be here to change my problem limit.

Another thing, I don't get any incoming connection

Can you connect to your own server?

****Yes, I’m well connected, here are my variables:

~ $ electrumx_rpc getinfo
{
"coin": "BitcoinSegwit",
"daemon": "localhost:8332/",
"daemon height": 757272,
"db height": 757272,
"db_flush_count": 2757,
"groups": 0,
"history cache": "0 lookups 0 hits 0 entries",
"merkle cache": "0 lookups 0 hits 0 entries",
"peers": {
"bad": 1,
"good": 115,
"never": 0,
"stale": 0,
"total": 116
},
"pid": 4243,
"request counts": {
"blockchain.block.header": 1,
"blockchain.headers.subscribe": 4,
"getinfo": 1,
"server.features": 4,
"server.peers.subscribe": 4,
"server.version": 4
},
"request total": 18,
"sessions": {
"count": 1,
"count with subs": 0,
"errors": 0,
"logged": 0,
"pending requests": 1,
"subs": 0
},
"tx hashes cache": "0 lookups 0 hits 0 entries",
"txs sent": 0,
"uptime": "01h 16m 08s",
"version": "ElectrumX 1.16.0"
`

In your log you have:

INFO:SessionManager:advertising service ssl://<IP_ADDRESS>:50002

I see electrumx is configured as:
export PUBLIC_IP=$(curl -s http://whatismyip.akamai.com/)
export REPORT_SERVICES=ssl://$PUBLIC_IP:50002

However, my client cannot connect to <IP_ADDRESS>:50002.

Ok thank you, yes I noticed that clients can’t connect to me.
Do I change by:
REPORT_SERVICS=ssl://<IP_ADDRESS>:50002 ?

Can you connect to your own server?

*Yes, I’m well connected

and now you say:

Ok thank you, yes I noticed that clients can’t connect to me.

... confusing. Not sure what you meant originally, but the answer then should have been "no".

Do I change by:
REPORT_SERVICS=ssl://<IP_ADDRESS>:50002 ?
And after that, can I delete the variable?:
export PUBLIC_IP=$(curl -s http://whatismyip.akamai.com/)

No, that won't work. You need an open port that you can listen on.
Typically you would either need a publicly routable IP address, or at least some service that has one and can forward a port to you. Alternatively you can run a Tor .onion service.

Can you connect to your own server?

*Yes, I’m well connected

and now you say:

Ok thank you, yes I noticed that clients can’t connect to me.

... confusing. Not sure what you meant originally, but the answer then should have been "no".

Do I change by:

REPORT_SERVICS=ssl://<IP_ADDRESS>:50002 ?

And after that, can I delete the variable?:

export PUBLIC_IP=$(curl -s http://whatismyip.akamai.com/)

No, that won't work. You need an open port that you can listen on.
Typically you would either need a publicly routable IP address, or at least some service that has one and can forward a port to you. Alternatively you can run a Tor .onion service.

Port 50002 is open.

I've tested now with $ telnet <IP_ADDRESS> 50002, and that port is not open.
I think you might be confused about external/routable IPs. Closing as there is no bug here.

Problem solved thank you very much for your help, I’m really sorry:
The port was well open but only for my IP address 🤦♂. I changed this and immediately saw incoming connections 😀, it works properly.

Hi,
I have electrumX, btc full node (incom10/10output) and an LND node with channel on a raspberry4 8go hdd 2to, graphic version (from time to time I need to access the internet from the raspberry).
How much RAM do I have to assign to electrumX for it to work properly please?
I assigned 4500mo to btc node.
Do you have any other tips to optimize my ram and cpu?
Thanks again


Support independent pods for each workflow/job?

[x] create separate pods for each workflow/job?
[ ] shared volume per workflow
[ ] publish outputs after a job finishes
[ ] make job template configurable (init containers, volumes, ...)

Hi @jirikuncar,
the individual jobs of a workflow are already one pod each. Currently only with a single ctr, but it's just a matter of exposing init containers. The jobs get a shared volume mounted by default.
We are working on a feature, where the submitted workflow engine instances (i.e. the thing that submits the steps in the right sequence) also gets started as either a) a Job for noninteractive workflows or b) a Deployment for interactive workflows (think: a notebook that steers the workflow)
init containers are something we have on our radar but it's not clear yet how to incorporate them. Generally I personally think the job-ctrl should be a quite thin layer on top of k8s and expose most APIs as long as they are secure.

@lukasheinrich, can the jobs then use the shared volume mount for sharing results? I thought results of individual jobs get pushed to CephFS.

yes in the case of a shared filesystem there is one CephFS volume that needs to be shared between the
  +-----[ engine ]
  |
+---+
| V  |----[ job1 ]
+---+
  |
  +[ job2 ]

Later (when jobs 1 and 2 are finished, the engine starts job 3):

  +-----[ engine ]
  |
+---+
| V |
+---+
  |
  +[ job3 ]

the volume lifetime is in principle independent of the pod lifetime (though one could also instruct the engine to copy only a subset of the data to a "results" volume and then delete the workflow volume, in which caase the lifetime of the volume and the engine pod is the same

Workflow pods are now independent and belong to the user who created them.


if we change the two image of order, we got the different size of matching-poing, ps : config, extractor:num_kpt=300

thanks

Hi,
Due to the concate operation in our network, the obtained matches are expected to differ when reversing img order. However, statistically the result should make very little differences.

can you eliminate the differences?

As a workaround, you may run the network twice for both img orders and find the intersection set, as is done for some neural outlier rejector(filtering based methods)


Better accessibility

Great component, however it is not accessible, thus we cannot use it. Would love to contribute, is this of interest?

@helgefredheim yes we want to make it accessible Would be great if you can contribute !!


Support for upcoming opam-monorepo release 0.3

The opam-monorepo developers are planning to release 0.3.0 soon, as such it seems reasonable to add support for it in ocaml-ci.
This PR consists of two parts:

Adding support for the 0.3 lockfile format and selecting the right version constraint (opam-monorepo 0.3.0 will not support the 0.2 format due to changes how dependencies are encoded - the migration is trivial and the tool notifies the user in that case)
Add support for installing dependencies that are supposed to be installed via opam which is a new feature in opam-monorepo starting at 0.3.0

That would be great because indeed we would like to try it first with opam-monorepo where we have a branch that uses its new version.
On April 18, 2022 6:29:15 AM GMT+02:00, Tim McGilchrist @.> wrote:
@. commented on this pull request.

Looks good.
I can do a test deploy if you have a project or two that use this new version @Leonidas-from-XIV
--
Reply to this email directly or view it on GitHub:
https://github.com/ocurrent/ocaml-ci/pull/445#pullrequestreview-944063736
You are receiving this because you were mentioned.
Message ID: @.***>

Since opam-monorepo 0.3.0 was released and merged to opam-repository I removed the pins from this PR. I think it should be ready to go.


Add name to peer review given

Adds Name to peer_review_given (for easier sorting and comparison)

please let @EricaHill know when update is available :)


Support additional numpad keys

Adds support for the Decimal, Equals and Enter numpad keys.

Looks great! Thanks for the PR! I'll try to cut a new release with your work soon.

Version 2.1 of device_query has been released that has this and the function key PR in it. Thanks for the work!


[SPARK-13223] [ML] Add stratified sampling to ML feature engineering

jira: https://issues.apache.org/jira/browse/SPARK-13223
I found it useful to add an sampling transformer during a case of fraud detection. It can be used in resampling or overSampling, which in turn is required by ensemble and unbalanced data processing.
Internally, it invoke the sampleByKey in Pair RDD operation.
Ideally, it should take the fraction parameter as Map[Any, Double], yet that would bring difficulty in model peresistence. Appreciate if anyone can show how to save Map[Any, Double] to a DataFrame.

Test build #50867 has started for PR 11102 at commit 022c836.

Test build #50867 has finished for PR 11102 at commit 022c836.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
 class StratifiedSamplingWriter(instance: StratifiedSampling) extends MLWriter

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50867/
Test FAILed.

@hhbyyh I think this is good to add. However, we need to think about the behavior during prediction. Certainly we don't want to apply sampling during prediction, but this is not addressed by the current pipeline implementation. Not for this PR, I just want to collect some ideas about this issue.

@mengxr Thanks for the review. Sorry for the late response, I was on a flight.
It's great to know DataFrame.stat.sampleB. One concern is that it does not allow withReplacement = true. That means oversampling is not supported.
For prediction, are you worrying about that users need to use the same PipelineModel for training and evaluation dataset? I would propose the solution to explicitly allow stepwise enable/disable on each stage for a PipelineModel. Thus users can skip specific steps in a pipeline.

Test build #51239 has started for PR 11102 at commit 78c80d7.

Test build #51239 has finished for PR 11102 at commit 78c80d7.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51239/
Test FAILed.

Test build #51260 has started for PR 11102 at commit 853fb96.

Test build #51260 has finished for PR 11102 at commit 853fb96.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51260/
Test PASSed.

Test build #51276 has started for PR 11102 at commit 525a80d.

Test build #51276 has finished for PR 11102 at commit 525a80d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51276/
Test PASSed.

@mengxr Appreciate if you have time for this

I tried to make it support multiple data types. As shown in the code, the main constraint is the save/load implementation. It'll be great if we can support Any key in the save/load.

Test build #55728 has started for PR 11102 at commit 8f8e747.

Test build #55728 has finished for PR 11102 at commit 8f8e747.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55728/
Test PASSed.

Close it since it's been overlooked for some time and can be implemented with https://github.com/apache/spark/pull/17583 easily. Thanks for the review and comments.


explore 'AST fingerprinting' for module/function identification (eg. to assist smart / stable renames, etc)

Another area I started looking into (but haven't deeply explored yet) for both figuring out how to map variable names to sections of code in a 'smart' way, and potentially also for module identification (see #41); is in the space of 'structural AST fingerprinting' or 'code similarity' algorithms and similar. (I realise that this is a rather deep/esoteric angle to be looking at this from, and that there are likely going to be far simpler/easier ways to implement the variable mapping/module identification in a 'good enough' way without going to this level of depth; but I'm curious to explore it regardless, to see if any good ideas come out of it)
I haven't gotten too far in my reading yet (got distracted on other things), but the high level of my idea was that maybe we could generate an 'AST fingerprint' that isn't impacted by the variable/function/etc names ('symbols') changing during minification; and then use that as the basis for the 'key' in the 'mappings file'; as that fingerprint could theoretically still identify a 'scope' (which might be a literal JS scope, or might be a higher level abstraction that we decide makes sense; the most abstract being probably at the bundled module level) even if the bundler decides to move some functions around to a different module/etc. Then obviously if we were able to generate those 'resilient fingerprints' to identify code even when it's been minified, that would make perfect sense to apply to module detection/etc (see #41) as well.
Some of the high level ideas / search terms that I was using to start my research in that area was things like:

AST fingerprinting
Source code similarity fingerprinting
Control flow graphs
Call flow graphs
Program dependence graph
etc

Here is a link dump of a bunch of the tabs I have open but haven't got around to reviewing in depth yet, RE: 'AST fingerprinting' / Code Similarity / etc:

Unsorted/Unreviewed Initial Link Dump RE: 'AST fingerprinting' / Code Similarity

https://openreview.net/forum?id=BJxWx0NYPr

On the Effectiveness of Program Analysis in AI and Vice Versa

https://dl.acm.org/doi/10.1145/3486860

A Unified Framework for Code Representation Learning

https://inria.hal.science/hal-01648996/document

A Survey of Program Slicing Techniques

https://hal.science/hal-00627811/document

From Program Slicing to Abstract Interpretation

https://igm.univ-mlv.fr/~chilowi/research/syntax_tree_fingerprinting/syntax_tree_fingerprinting_ICPC09.pdf

Syntax Tree Fingerprinting for Source Code Similarity Detection

https://ieeexplore.ieee.org/document/9960266

Exploring the Landscape of Deep Learning for Source Code Analysis

https://en.wikipedia.org/wiki/Program_dependence_graph

Program Dependence Graph - Wikipedia

https://digitalcommons.calpoly.edu/cgi/viewcontent.cgi?article=3437&context=theses

Enhancing Software Reliability Using Program Analysis Techniques

https://dl.acm.org/doi/pdf/10.1145/1286821.1286826

Static Analysis of Executables to Detect Malicious Patterns

https://patents.google.com/patent/US9459861B1/en

US Patent US9459861B1: Techniques for Code Analysis and Optimization

https://www.unomaha.edu/college-of-information-science-and-technology/research-labs/_files/software-nsf.pdf

Software Engineering Research and Practice: NSF Workshop Report

https://www.computer.org/csdl/journal/ts/2023/08/10125077/1Nc4Vd4vb7W

Advances in Automated Program Repair: IEEE Transactions

https://stackoverflow.com/questions/7283702/assembly-level-function-fingerprint

Stack Overflow: Assembly-Level Function Fingerprinting

https://en.wikipedia.org/wiki/Control-flow_graph

Control-Flow Graph - Wikipedia

https://www.kapravelos.com/publications/fptechniques-www23.pdf

Novel Techniques for Malware and Vulnerability Analysis in Web Applications

https://pubmed.ncbi.nlm.nih.gov/12197666/

PubMed: Program Analysis Techniques for Software Debugging

https://stackoverflow.com/questions/15087195/data-flow-graph-construction

Stack Overflow: Data Flow Graph Construction in Programming

https://www.cs.columbia.edu/~suman/secure_sw_devel/Basic_Program_Analysis_CF.pdf

Basic Program Analysis Techniques for Secure Software Development

https://codeql.github.com/docs/writing-codeql-queries/about-data-flow-analysis/

CodeQL Guide: Understanding Data Flow Analysis for Secure Software Development

https://codereview.stackexchange.com/questions/276387/call-flow-graph-from-python-abstract-syntax-tree

Code Review Stack Exchange: Call Flow Graph from Python AST

https://clang.llvm.org/docs/DataFlowAnalysisIntro.html

LLVM Clang Documentation: Introduction to Data Flow Analysis

https://www.researchgate.net/figure/AST-based-program-graph-with-additional-control-and-data-flow-edges_fig2_370980383

ResearchGate: AST-Based Program Graphs with Control and Data Flow Edges

Originally posted by @0xdevalias in https://github.com/pionxzh/wakaru/issues/34#issuecomment-1843850057

See Also

https://github.com/pionxzh/wakaru/issues/34
https://github.com/pionxzh/wakaru/issues/73

Have been spending some more time in binary reverse engineering land lately, and stumbled across this tool (Diaphora). While it's focus is on binary reverse engineering, some of the features it mentioned sounded like they would be interesting/useful to look deeper into for this 'AST Fingerprinting' sort of idea, eg.

Porting symbol names and comments
Similarity ratio calculation
Call graph matching calculation
Dozens of heuristics based on graph theory, assembler, bytes, functions' features, etc...
Pseudo-code based heuristics

There might be some ideas/patterns/algorithms/similar that we could use from there for implementing AST fingerprinting on JS code.

http://diaphora.re/

Diaphora
A Free and Open Source Program Diffing Tool

Diaphora (διαφορά, Greek for 'difference') version 3.0 is the most advanced program diffing tool (working as an IDA plugin) available as of today (2023). It was released first during SyScan 2015 and has been actively maintained since this year: it has been ported to every single minor version of IDA since 6.8 to 8.3.
Diaphora supports versions of IDA >= 7.4 because the code only runs in Python 3.X (Python 3.11 was the last version being tested).

https://github.com/joxeankoret/diaphora

Diaphora, the most advanced Free and Open Source program diffing tool.

Diaphora has many of the most common program diffing (bindiffing) features you might expect, like:

Diffing assembler.
Diffing control flow graphs.
Porting symbol names and comments.
Adding manual matches.
Similarity ratio calculation.
Batch automation.
Call graph matching calculation.
Dozens of heuristics based on graph theory, assembler, bytes, functions' features, etc...

However, Diaphora has also many features that are unique, not available in any other public tool. The following is a non extensive list of unique features:

Ability to port structs, enums, unions and typedefs.
Potentially fixed vulnerabilities detection for patch diffing sessions.
Support for compilation units (finding and diffing compilation units).
Microcode support.
Parallel diffing.
Pseudo-code based heuristics.
Pseudo-code patches generation.
Diffing pseudo-codes (with syntax highlighting!).
Scripting support (for both the exporting and diffing processes).

https://github.com/FernandoDoming/r2diaphora

r2diaphora
r2diaphora is a port of Diaphora to radare2 and MariaDB. It also uses r2ghidra as decompiler by default, with support for other decompilers such as pdc.

Port of the binary diffing library, diaphora, for radare2 and mariadb

The Stack Graph / Scope Graph links/references I shared in https://github.com/pionxzh/wakaru/issues/34#issuecomment-2035859278 may be relevant to this issue as well.

Some more 'prior art' from the binary reverse engineering world:

https://hex-rays.com/products/ida/tech/flirt/in_depth/

IDA F.L.I.R.T. Technology: In-Depth

One major stumbling block in the disassembly of programs written in modern high
level languages is the time required to isolate library functions.

To assist IDA users we attempted to create an algorithm to recognize the
standard library functions.

The idea
To address those issues, we created a database of all the functions from all libraries we wanted to recognize. IDA now checks, at each byte of the program being disassembled, whether this byte can mark the start of a standard library function.
The information required by the recognition algorithm is kept in a signature file. Each function is represented by a pattern. Patterns are first 32 bytes of a function where all variant bytes are marked.
..snip..

Sequences of bytes are kept in the nodes of the tree.

When two functions have the same first 32 bytes, they are stored in the same leaf of the tree. To resolve that situation, we calculate the CRC16 of the bytes starting from position 33 until till the first variant byte. The CRC is stored in the signature file. The number of bytes used to calculate that CRC also needs to be saved, as it differs from function to function.

etc

While the exact specifics of that method won't be relevant here (since we're operating on JS, and not raw bytes); some of the more general concepts might be.
Interestingly, that ends up being a more refined version of some binary offset finding code I wrote for another project:

https://github.com/0xdevalias/poc-re-binsearch

Proof of Concept (PoC) code/notes exploring reverse engineering techniques for macOS fat binaries, focusing on binary searching and automatic offset identification

https://github.com/0xdevalias/poc-re-binsearch/blob/main/find_bytes.py
https://github.com/0xdevalias/poc-re-binsearch/blob/main/find_fat_binary_offsets.py
https://github.com/0xdevalias/poc-re-binsearch/blob/main/diff_hex.py

I've been thinking about this topic and found this repo when searching for if someone had done it before and/or for a debundler to build it on top of. The basic idea I was imagining was:

First perform some basic normalization of the code
Rename all local variables according to some deterministic scheme
Calculate a hash of the leaf modules that are not referring to any other module (either hash the AST or the normalized source, both should work) and rename these modules according to their hash, so they are content-addressed
Keep traversing the graph and calculating the hashes of modules where all their dependencies' names have been replaced with hashes

After this we can lookup the hashes in a database of known hashes for library functions. If we have information on which libraries may have been used, creating this database should be fairly simple.
This approach doesn't do any kind of fuzzy matching, but as long as the normalization works well enough and the output doesn't vary in too many ways that are difficult to normalize away depending on e.g. bundler config, it should be fairly reliable.
The approach is kind of similar to how the content addressed programming language Unison does their hashing.
If we want to allow more fine-grained fingerprinting we could use some kind of De Bruijn index instead for the local variables, so local snippets would have the same variable names regardless of their context. This wouldn't produce valid JS code, but that doesn't matter since the result is only used for hashing, not for output. But I think focusing on just entire modules would be a good start and give much value.

I've been thinking about this topic and found this repo when searching for if someone had done it before and/or for a debundler to build it on top of.

@anka-213 Curious (if you're open to/able to share), what your use case for this sort of thing would be?

The basic idea I was imagining was:

First perform some basic normalization of the code
Rename all local variables according to some deterministic scheme

This approach doesn't do any kind of fuzzy matching, but as long as the normalization works well enough and the output doesn't vary in too many ways that are difficult to normalize away depending on e.g. bundler config, it should be fairly reliable.

@anka-213 This basically aligns with one of the ways of how I was thinking it would probably work at a high level as well; though I think the key/crux of it would be figuring out the normalisation (including stabilising or not including variable/function identifiers that churn) in a way that is resilient to all the 'optimisations' a bundler/minifier might choose to make.
That may mean that it would need to run on 'partially unminified' code, though in the ideal case, it should be able to work with as little 'pre-processing' of the minified code as possible; as this module identification would be used as part of the unminification process (for certain aspects).

The approach is kind of similar to how the content addressed programming language Unison does their hashing.

@anka-213 Just had a read through that blog, and it sounds like a really interesting approach!

If we want to allow more fine-grained fingerprinting we could use some kind of De Bruijn index instead for the local variables, so local snippets would have the same variable names regardless of their context. This wouldn't produce valid JS code, but that doesn't matter since the result is only used for hashing, not for output.

@anka-213 I only quickly skimmed the wiki pages for De Bruijn index / De Bruijn notation, so I might not be grasping it fully, but from what I saw, it seems like you could probably model it in a way that would fit the semantics to produce valid JS variable names/code still.

Another method (that I can't remember if I've ever written out in full here) is somewhat based on the more manual approach I was taking at one point:

that can help us transform the code and give the extracted module a better name other than module-xxxx.js

This could then also tie in well with some of the ideas for 'unmangling identifiers' that I laid out here:

https://github.com/pionxzh/wakaru/issues/34#issuecomment-1807393509

Theoretically if we can identify a common open source module, we could also have pre-processed that module to extract variable/function names, that we could then potentially apply back to the identified module.
I kind of think of this like 'debug symbols' used in compiled binaries.
Though technically, if you know the module and can get the original source; and you know the webpacked version of that code; you could also generate a sourcemap that lets the user map between the 2 versions of the code.

When I was manually attempting to reverse and identify the modules in #40, a couple of techniques I found useful:

searching for Symbol()s
searching for React .displayName and similar
searching for other arrays of static strings/similar
once interesting candidates had been found, searching for them on GitHub code search to try and identify the library/narrow things down

Edit: This might not be useful right now, but just added a new section to one of my gists with some higher level notes/thoughts on fingerprinting modules; that I might expand either directly, or based on how this issue pans out:

https://gist.github.com/0xdevalias/d8b743efb82c0e9406fc69da0d6c6581#file-fingerprinting-minified-javascript-libraries-md

While it might be more effort than it's worth, it may also be possible to extract the patterns that wappalyzer was using to identify various libraries; which I made some basic notes on in this revision to the above gist:

https://gist.github.com/0xdevalias/d8b743efb82c0e9406fc69da0d6c6581/revisions#diff-3a443fd1fa55e0ff7e0206791ae1fb8bbe16f7f1ae5505998cbafb38318f8e69R381-R441

Originally posted by @0xdevalias in https://github.com/pionxzh/wakaru/issues/41#issuecomment-1810097408

Specifically, identifying the types of things that are usually not minified/mangled by a bundler/minifier (Symbol()s, React's .displayName, static strings, etc; and using those parts as the 'source' for the fingerprint/similar. In a way, I guess this could also be considered a type of normalisation.
One benefit of this approach, is that those same 'key identifiers' can be used with GitHub Code search or similar tools to help narrow down and identify an otherwise unknown module/library. This could even probably be partially automated using the GitHub API; and then provide an easy way for users to contribute the relevant details/hash/etc for an identified module back to the 'core database' (in a similar way to how nmap allows users to submit service fingerprints (Ref: 1, 2)
Here is some further 'prior art' from a tool that seems to use this sort of method to target the functions it wants to interact with:

This specific implementation is more related to detecting and injecting into webpack modules at runtime, but it might have some useful ideas/concepts that are applicable at the AST level too:

https://github.com/GooseMod/GooseMod/blob/master/src/util/discord/webpackModules.js

// ..snip..

export const common = { // Common modules
  React: findByProps('createElement'),
  ReactDOM: findByProps('render', 'hydrate'),

  Flux: findByProps('Store', 'connectStores'),
  FluxDispatcher: findByProps('register', 'wait'),

  i18n: findByProps('Messages', '_requestedLocale'),

  channels: findByProps('getChannelId', 'getVoiceChannelId'),
  constants: findByProps('API_HOST')
};

Originally posted by @0xdevalias in https://github.com/pionxzh/wakaru/issues/41#issuecomment-1890296652

This is potentially more of a generalised/'naive' approach to the problem, but it would also be interesting to see if/how well an embedding model tuned for code would do at solving this sort of problem space:

https://openai.com/blog/introducing-text-and-code-embeddings

https://platform.openai.com/docs/guides/embeddings

An embedding is a vector (list) of floating point numbers. The distance between two vectors measures their relatedness. Small distances suggest high relatedness and large distances suggest low relatedness.

https://platform.openai.com/docs/api-reference/embeddings

https://engineering.fb.com/2017/03/29/data-infrastructure/faiss-a-library-for-efficient-similarity-search/

Faiss: A library for efficient similarity search

Also, here's the latest version of my open tabs 'reading list' in this space of things, in case any of it is relevant/interesting/useful here:

Unsorted/Unreviewed Link Dump RE: 'AST fingerprinting' / Code Similarity (v2)

https://en.wikipedia.org/wiki/Content_similarity_detection

Content similarity detection

https://arxiv.org/abs/2306.16171

A systematic literature review on source code similarity measurement and clone detection: techniques, applications, and challenges (2023)

Measuring and evaluating source code similarity is a fundamental software engineering activity that embraces a broad range of applications, including but not limited to code recommendation, duplicate code, plagiarism, malware, and smell detection. This paper proposes a systematic literature review and meta-analysis on code similarity measurement and evaluation techniques to shed light on the existing approaches and their characteristics in different applications. We initially found over 10000 articles by querying four digital libraries and ended up with 136 primary studies in the field. The studies were classified according to their methodology, programming languages, datasets, tools, and applications. A deep investigation reveals 80 software tools, working with eight different techniques on five application domains. Nearly 49% of the tools work on Java programs and 37% support C and C++, while there is no support for many programming languages. A noteworthy point was the existence of 12 datasets related to source code similarity measurement and duplicate codes, of which only eight datasets were publicly accessible. The lack of reliable datasets, empirical evaluations, hybrid methods, and focuses on multi-paradigm languages are the main challenges in the field. Emerging applications of code similarity measurement concentrate on the development phase in addition to the maintenance.

https://link.springer.com/article/10.1007/s10664-017-9564-7

A comparison of code similarity analysers (2017)

Copying and pasting of source code is a common activity in software engineering. Often, the code is not copied as it is and it may be modified for various purposes; e.g. refactoring, bug fixing, or even software plagiarism. These code modifications could affect the performance of code similarity analysers including code clone and plagiarism detectors to some certain degree. We are interested in two types of code modification in this study: pervasive modifications, i.e. transformations that may have a global effect, and local modifications, i.e. code changes that are contained in a single method or code block. We evaluate 30 code similarity detection techniques and tools using five experimental scenarios for Java source code. These are (1) pervasively modified code, created with tools for source code and bytecode obfuscation, and boiler-plate code, (2) source code normalisation through compilation and decompilation using different decompilers, (3) reuse of optimal configurations over different data sets, (4) tool evaluation using ranked-based measures, and (5) local + global code modifications. Our experimental results show that in the presence of pervasive modifications, some of the general textual similarity measures can offer similar performance to specialised code similarity tools, whilst in the presence of boiler-plate code, highly specialised source code similarity detection techniques and tools outperform textual similarity measures. Our study strongly validates the use of compilation/decompilation as a normalisation technique. Its use reduced false classifications to zero for three of the tools. Moreover, we demonstrate that optimal configurations are very sensitive to a specific data set. After directly applying optimal configurations derived from one data set to another, the tools perform poorly on the new data set. The code similarity analysers are thoroughly evaluated not only based on several well-known pair-based and query-based error measures but also on each specific type of pervasive code modification. This broad, thorough study is the largest in existence and potentially an invaluable guide for future users of similarity detection in source code.

https://www.researchgate.net/publication/2840981_Winnowing_Local_Algorithms_for_Document_Fingerprinting

Winnowing: Local Algorithms for Document Fingerprinting (2003)

Digital content is for copying: quotation, revision, plagiarism, and file sharing all create copies. Document fingerprinting is concerned with accurately identifying copying, including small partial copies, within large sets of documents. We introduce the class of local document fingerprinting algorithms, which seems to capture an essential property of any fingerprinting technique guaranteed to detect copies. We prove a novel lower bound on the performance of any local algorithm. We also develop winnowing, an efficient local fingerprinting algorithm, and show that winnowing's performance is within 33% of the lower bound. Finally, we also give experimental results on Web data, and report experience with Moss, a widely-used plagiarism detection service.

https://theory.stanford.edu/~aiken/publications/papers/sigmod03.pdf

https://www.researchgate.net/publication/375651686_Source_Code_Plagiarism_Detection_with_Pre-Trained_Model_Embeddings_and_Automated_Machine_Learning

Source Code Plagiarism Detection with Pre-Trained Model Embeddings and Automated Machine Learning (2023)

https://aclanthology.org/2023.ranlp-1.34.pdf

https://www.researchgate.net/publication/262322336_A_Source_Code_Similarity_System_for_Plagiarism_Detection

A Source Code Similarity System for Plagiarism Detection (2013)

Source code plagiarism is an easy to do task, but very difficult to detect without proper tool support. Various source code similarity detection systems have been developed to help detect source code plagiarism. Those systems need to recognize a number of lexical and structural source code modifications. For example, by some structural modifications (e.g. modification of control structures, modification of data structures or structural redesign of source code) the source code can be changed in such a way that it almost looks genuine. Most of the existing source code similarity detection systems can be confused when these structural modifications have been applied to the original source code. To be considered effective, a source code similarity detection system must address these issues. To address them, we designed and developed the source code similarity system for plagiarism detection. To demonstrate that the proposed system has the desired effectiveness, we performed a well-known conformism test. The proposed system showed promising results as compared with the JPlag system in detecting source code similarity when various lexical or structural modifications are applied to plagiarized code. As a confirmation of these results, an independent samples t-test revealed that there was a statistically significant difference between average values of F-measures for the test sets that we used and for the experiments that we have done in the practically usable range of cut-off threshold values of 35–70%.

https://www.mdpi.com/2076-3417/10/21/7519

A Source Code Similarity Based on Siamese Neural Network (2020)

Finding similar code snippets is a fundamental task in the field of software engineering.
Several approaches have been proposed for this task by using statistical language model which focuses on syntax and structure of codes rather than deep semantic information underlying codes. In this paper, a Siamese Neural Network is proposed that maps codes into continuous space vectors and try to capture their semantic meaning. Firstly, an unsupervised pre-trained method that models code snippets as a weighted series of word vectors. The weights of the series are fitted by the Term Frequency-Inverse Document Frequency (TF-IDF). Then, a Siamese
Neural Network trained model is constructed to learn semantic vector representation of code snippets. Finally, the cosine similarity is provided to measure the similarity score between pairs of code snippets. Moreover, we have implemented our approach on a dataset of functionally similar code. The experimental results show that our method improves some performance over single word embedding method.

https://www.researchgate.net/publication/337196468_Detecting_Source_Code_Similarity_Using_Compression

Detecting Source Code Similarity Using Compression (2019)

Different forms of plagiarism make a fair assessment of student assignments more difficult. Source code plagiarisms pose a significant challenge especially for automated assessment systems aimed for students' programming solutions. Different automated assessment systems employ different text or source code similarity detection tools, and all of these tools have their advantages and disadvantages. In this paper, we revitalize the idea of similarity detection based on string complexity and compression. We slightly adapt an existing, third-party, approach, implement it and evaluate its potential on synthetically generated cases and on a small set of real student solutions. On synthetic cases, we showed that average deviation (in absolute values) from the expected similarity is less than 1% (0.94%). On the real-life examples of student programming solutions we compare our results with those of two established tools. The average difference is around 18.1% and 11.6%, while the average difference between those two tools is 10.8%. However, the results of all three tools follow the same trend. Finally, a deviation to some extent is expected as observed tools apply different approaches that are sensitive to other factors of similarities. Gained results additionally demonstrate open challenges in the field.

https://ceur-ws.org/Vol-2508/paper-pri.pdf

https://www.nature.com/articles/s41598-023-42769-9

Binary code similarity analysis based on naming function and common vector space (2023)

Binary code similarity analysis is widely used in the field of vulnerability search where source code may not be available to detect whether two binary functions are similar or not. Based on deep learning and natural processing techniques, several approaches have been proposed to perform cross-platform binary code similarity analysis using control flow graphs. However, existing schemes suffer from the shortcomings of large differences in instruction syntaxes across different target platforms, inability to align control flow graph nodes, and less introduction of high-level semantics of stability, which pose challenges for identifying similar computations between binary functions of different platforms generated from the same source code. We argue that extracting stable, platform-independent semantics can improve model accuracy, and a cross-platform binary function similarity comparison model N_Match is proposed. The model elevates different platform instructions to the same semantic space to shield their underlying platform instruction differences, uses graph embedding technology to learn the stability semantics of neighbors, extracts high-level knowledge of naming function to alleviate the differences brought about by cross-platform and cross-optimization levels, and combines the stable graph structure as well as the stable, platform-independent API knowledge of naming function to represent the final semantics of functions. The experimental results show that the model accuracy of N_Match outperforms the baseline model in terms of cross-platform, cross-optimization level, and industrial scenarios. In the vulnerability search experiment, N_Match significantly improves hit@N, the mAP exceeds the current graph embedding model by 66%. In addition, we also give several interesting observations from the experiments. The code and model are publicly available at https://www.github.com/CSecurityZhongYuan/Binary-Name_Match

https://arxiv.org/abs/2305.03843

REINFOREST: Reinforcing Semantic Code Similarity for Cross-Lingual Code Search Models (2023)

This paper introduces a novel code-to-code search technique that enhances the performance of Large Language Models (LLMs) by including both static and dynamic features as well as utilizing both similar and dissimilar examples during training. We present the first-ever code search method that encodes dynamic runtime information during training without the need to execute either the corpus under search or the search query at inference time and the first code search technique that trains on both positive and negative reference samples. To validate the efficacy of our approach, we perform a set of studies demonstrating the capability of enhanced LLMs to perform cross-language code-to-code search. Our evaluation demonstrates that the effectiveness of our approach is consistent across various model architectures and programming languages. We outperform the state-of-the-art cross-language search tool by up to 44.7%. Moreover, our ablation studies reveal that even a single positive and negative reference sample in the training process results in substantial performance improvements demonstrating both similar and dissimilar references are important parts of code search. Importantly, we show that enhanced well-crafted, fine-tuned models consistently outperform enhanced larger modern LLMs without fine tuning, even when enhancing the largest available LLMs highlighting the importance for open-sourced models. To ensure the reproducibility and extensibility of our research, we present an open-sourced implementation of our tool and training procedures called REINFOREST.

https://www.usenix.org/conference/usenixsecurity21/presentation/ahmadi

Finding Bugs Using Your Own Code: Detecting Functionally-similar yet Inconsistent Code (2021)

Probabilistic classification has shown success in detecting known types of software bugs. However, the works following this approach tend to require a large amount of specimens to train their models. We present a new machine learning-based bug detection technique that does not require any external code or samples for training. Instead, our technique learns from the very codebase on which the bug detection is performed, and therefore, obviates the need for the cumbersome task of gathering and cleansing training samples (e.g., buggy code of certain kinds). The key idea behind our technique is a novel two-step clustering process applied on a given codebase. This clustering process identifies code snippets in a project that are functionally-similar yet appear in inconsistent forms. Such inconsistencies are found to cause a wide range of bugs, anything from missing checks to unsafe type conversions. Unlike previous works, our technique is generic and not specific to one type of inconsistency or bug. We prototyped our technique and evaluated it using 5 popular open source software, including QEMU and OpenSSL. With a minimal amount of manual analysis on the inconsistencies detected by our tool, we discovered 22 new unique bugs, despite the fact that many of these programs are constantly undergoing bug scans and new bugs in them are believed to be rare.

https://www.usenix.org/system/files/sec21summer_ahmadi.pdf

https://theory.stanford.edu/~aiken/moss/

MOSS: A1 System for Detecting Software Similarity

https://github.com/fanghon/antiplag

antiplag similarity checking software for program codes, documents, and pictures
The software mainly checks and compares the similarities between electronic assignments submitted by students. It can detect the similarities between electronic assignments submitted by students and can analyze the content of multiple programming languages (such as java, c/c++, python, etc.) and multiple formats (txt, doc, docx, pdf, etc.) Comparative analysis of text and image similarities in multiple formats (png, jpg, gif, bmp, etc.) between English and simplified and traditional Chinese documents, and output codes, texts, and images with high similarity, thereby helping to detect plagiarism between students. the behavior of.

https://github.com/dodona-edu/dolos

Dolos
Dolos is a source code plagiarism detection tool for programming exercises. Dolos helps teachers in discovering students sharing solutions, even if they are modified. By providing interactive visualizations, Dolos can also be used to sensitize students to prevent plagiarism.

https://dolos.ugent.be/
https://dolos.ugent.be/about/algorithm.html

How Dolos works
Conceptually, the plagiarism detection pipeline of Dolos can be split into four successive steps:

Tokenization
Fingerprinting
Indexing
Reporting

Tokenization
To be immune against masking plagiarism by techniques such as renaming variables and functions, Dolos doesn't directly process the source code under investigation. It starts by performing a tokenization step using Tree-sitter. Tree-sitter can generate syntax trees for many programming languages, converts source code to a more structured form, and masks specific naming of variables and functions.

Fingerprinting
To measure similarities between (converted) files, Dolos tries to find common sequences of tokens. More specifically, it uses subsequences of fixed length called k-grams. To efficiently make these comparisons and reduce the memory usage, all k-grams are hashed using a rolling hash function (the one used by Rabin-Karp in their string matching algorithm). The length k of k-grams can be with the -k option.
To further reduce the memory usage, only a subset of all hashes are stored. The selection of hashes is done by the Winnowing algorithm as described by (Schleimer, Wilkerson and Aiken). In short: only the hash with the smallest numerical value is kept for each window. The window length (in k-grams) can be altered with the -w option.
The remaining hashes are the fingerprints of the analyzed files. Internally, these are stored as simple integers.

Indexing
Because Dolos needs to compare all files with each other, it is more efficient to first create an index containing the fingerprints of all files. For each of the fingerprints encountered in any of the files, we store the file and the corresponding line number where we encountered that fingerprint.
As soon as a fingerprint is stored in the index twice, this is recorded as a match between the two files because they share at least one k-gram.

Reporting
Dolos finally collects all fingerprints that occur in more than one file and aggregates the results into a report.
This report contains all file pairs that have at least one common fingerprint, together with some metrics:

similarity: the fraction of shared fingerprints between the two files
total overlap: the absolute value of shared fingerprints, useful for larger projects
longest fragment: the length (in fingerprints) of the longest subsequence of fingerprints matching between the two files, useful when not the whole source code is copied

https://dolos.ugent.be/about/languages.html
https://dolos.ugent.be/about/publications.html

Publications
Dolos is developed by Team Dodona at Ghent University in Belgium. Our research is published in the following journals and conferences.

https://github.com/danielplohmann/mcrit

MinHash-based Code Relationship & Investigation Toolkit (MCRIT)
MCRIT is a framework created to simplify the application of the MinHash algorithm in the context of code similarity. It can be used to rapidly implement "shinglers", i.e. methods which encode properties of disassembled functions, to then be used for similarity estimation via the MinHash algorithm. It is tailored to work with disassembly reports emitted by SMDA.

https://github.com/BK-SCOSS/scoss

scoss
A Source Code Similarity System - SCOSS

https://github.com/island255/source2binary_dataset_construction

Source2binary Dataset Construction
This is the repository for the paper "One to One or One to many? What function inline brings to binary similarity analysis".

https://github.com/JackHCC/Pcode-Similarity

Pcode-Similarity
Algorithm for calculating similarity between function and library function.

https://github.com/JackHCC/Awesome-Binary-Code-Similarity-Detection-2021

Awesome Binary code similarity detection 2021
Awesome list for Binary Code Similarity Detection in 2021

https://github.com/Jaso1024/Semantic-Code-Embeddings

SCALE: Semantic Code Analysis via Learned Embeddings (2023)
3rd best paper on Artificial Intelligence track | presented at the 2023 International Conference on AI, Blockchain, Cloud Computing and Data Analytics
This repository holds the code and supplementary materials for SCALE: Semantic Code Analysis via Learned Embeddings. This research explores the efficacy of contrastive learning alongside large language models as a paradigm for developing a model capable of creating code embeddings indicative of code on a functional level.
Existing pre-trained models in NLP have demonstrated impressive success, surpassing previous benchmarks in various language-related tasks. However, when it comes to the field of code understanding, these models still face notable limitations. Code isomorphism, which deals with determining functional similarity between pieces of code, presents a challenging problem for NLP models. In this paper, we explore two approaches to code isomorphism. Our first approach, dubbed SCALE-FT, formulates the problem as a binary classification task, where we feed pairs of code snippets to a Large Language Model (LLM), using the embeddings to predict whether the given code segments are equivalent. The second approach, SCALE-CLR, adopts the SimCLR framework to generate embeddings for individual code snippets. By processing code samples with an LLM and observing the corresponding embeddings, we assess the similarity of two code snippets. These approaches enable us to leverage function-based code embeddings for various downstream tasks, such as code-optimization, code-comment alignment, and code classification. Our experiments on the CodeNet Python800 benchmark demonstrate promising results for both approaches. Notably, our SCALE-FT using Babbage-001 (GPT-3) achieves state-of-the-art performance, surpassing various benchmark models such as GPT-3.5 Turbo and GPT-4. Additionally, Salesforce's 350-million parameter CodeGen, when trained with the SCALE-FT framework, surpasses GPT-3.5 and GPT-4.

https://github.com/Aida-yy/binary-sim

binary similarity using Deep learning

Features: Function semantic information + control flow graph
Semantic feature extraction: extract the byte data, assembly instruction data, and integer data of the function respectively, use independent encoders (DPCNN, TextCNN) to encode the text representation, and obtain its Embedding representation.
Structural feature extraction, based on CFG and the assembly instructions in each block, generates ACFG, uses graph neural network to encode ACFG, and obtains Embedding representation; in addition, considering that the node order of the control flow graph of similar functions is also similar, the CFG's The adjacency matrix is taken as input and CNN is used to obtain its Embedding representation.
Contrastive learning model structure: InfoNCE loss + In-batch negatives

Order Matters: Semantic-Aware Neural Networks for Binary Code Similarity Detection
Investigating Graph Embedding Methods for Cross-Platform Binary Code Similarity Detection
SimCSE: Simple Contrastive Learning of Sentence Embeddings

https://github.com/jorge-martinez-gil/crosslingual-clone-detection

Transcending Language Barriers in Software Engineering with Crosslingual Code Clone Detection (2024)
Systematic study to determine the best methods to assess the similarity between code snippets in different programming languages

You can also find the first link dump of content in the collapsible in the first post on this issue.

A little more (speculative) 'prior art' from the binary reversing world:

https://binary.ninja/2022/06/20/introducing-tanto.html#potential-uses-and-some-speculation

What I’ve found most interesting, and have been speculating about, is using variable slices like these (though not directly through the UI) in the function fingerprinting space. I’ve long suspected that a dataflow-based approach to fingerprinting might prove to be robust against compiler optimizations and versions, as well as source code changes that don’t completely redefine the implementation of a function. Treating each variable slice as a record of what happens to data within a function, a similarity score for two slices could be generated from the count of matching operations, matching constant interactions (2 + var_a), and matching variable interactions (var_f + var_a). Considering all slices, a confidence metric could be derived for whether two functions match. Significant research would be required to answer these questions concretely… and, if you could solve subgraph isomorphism at the same time, that’d be great!

Further 'prior art', an example of an 'obfuscation detector' based on AST structure:

Here are projects that try to support many different ones: PerimeterX/restringer, ben-sb/javascript-deobfuscator

Instead I'd rather add more interactive actions that make manually working on unknown obfuscators faster and let the user decide if its safe

Linked from that restringer repo, I came across this project:

https://github.com/PerimeterX/obfuscation-detector

Detect different types of JS obfuscation by their AST structure

https://github.com/PerimeterX/obfuscation-detector#supported-obfuscation-types
https://github.com/PerimeterX/obfuscation-detector/tree/main/src/detectors

It could be cool to have a similar sort of 'obfuscation detector' feature within webcrack, particularly if it was paired with the 'interactive actions'. The 'detector' rules could suggest which obfuscations seem to be in place, and could then potentially recommend corresponding rules, etc.
Originally posted by @0xdevalias in https://github.com/j4k0xb/webcrack/issues/76#issuecomment-2116401646

There has recently been a new source of discussion around code fingerprinting and module identification over on the humanify repo in this issue:

https://github.com/jehna/humanify/issues/97


Pre-provisioned (and common) infrastructure for all participants (at an AWS event and using the workshop in their own account)

What would you like to be added?
Current state:
There is currently a method to pre-provision infrastructure that takes longer to create (using addon_infrastructure.tf), but this only helps participants who are using EKS Workshop as part of an AWS event.
Desired state:

A method to pre-provision infrastructure for all participants – regardless of whether they are using EKS Workshop at an AWS event or using their own AWS account.
(Related to point 1): A method for having common infrastructure that is available across all modules.

Why is this needed?
This is based on recent observations while creating the ‘Observability with OpenSearch’ sub-module.
Benefits of having pre-provisioned and common infrastructure for all participants (AWS event or in participants account):

Consistent participant experience: Provisioning or destroying an OpenSearch domain can take 20+ minutes. The user experience is not great for participants who are following EKS Workshop in their own account.  They will have to wait 20+ minutes for prepare-environment observability/opensearch to create the OpenSearch domain and also wait a similar amount of time when they run the next prepare-environment command to switch to the next lab, because the OpenSearch domain gets destroyed at that point.  Having a method to pre-provision infrastructure for all participants (AWS Event or in your own account) will address this.
Common infrastructure as an instructional tool: OpenSearch (and observability, more generally) can be useful for bringing to life some of the abstract concepts in EKS Workshop especially for participants who are new to Kubernetes.  For example, observability tools can show participants what is occurring inside the cluster during the autoscaling, networking or security modules of EKS Workshop.  Using OpenSearch observability as an instructional tool would require the associated infrastructure to be pre-provisioned in a consistent manner for all participants.

I think we could add this as an option. Its worth bearing in mind that one of the reasons we stopped doing this pre-provisioning is being cost-conscious on the part of customers so we don't have all this infrastructure running for potentially longer periods of time.


#737 - PMD fixes

Contains PMD fixed identified during the analysis profile executing of the project. These fixes are:

Putting guards around log statements
Correcting number of parameters to log statements
Collapsing if clauses

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] I have signed the Adobe Open Source CLA.
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.

If PMD checks are mandatory please enable the analysis profile by default (both for CI with GitHub Actions and for local Maven builds), at the same time the other profile analysisCI can probably be removed

@kwin this is in preparation to support the analysis profile. There are further issues as well in that profile that i need to resolve, but haven't gotten to.


Feat: Add Logo

Should resolve #24

@codito If you know how to do this, could you tell me what I did wrong? I can spend more time on it, but figured I would ask before I spend too much more time playing with it.

This could be because we're running on Visual Studio 2017 image in appveyor which has an older dotnet sdk. Can we try with Visual Studio 2019?

Ok, so that generated the same errors I see when I run locally. Will try to figure these out...

C:\Program Files\dotnet\sdk\3.1.401\Sdks\NuGet.Build.Tasks.Pack\build\NuGet.Build.Tasks.Pack.targets(198,5): error NU5127: This package does not contain a lib/ or ref/ folder, and will therefore be treated as compatible for all frameworks. Since framework specific files were found under the build/ directory for netstandard1.0, net45, consider creating the following empty files to correctly narrow the compatibility of the package: [C:\projects\junit-testlogger\src\package\package.csproj]
C:\Program Files\dotnet\sdk\3.1.401\Sdks\NuGet.Build.Tasks.Pack\build\NuGet.Build.Tasks.Pack.targets(198,5): error NU5127: -lib/netstandard1.0/. [C:\projects\junit-testlogger\src\package\package.csproj]
C:\Program Files\dotnet\sdk\3.1.401\Sdks\NuGet.Build.Tasks.Pack\build\NuGet.Build.Tasks.Pack.targets(198,5): error NU5127: -lib/net45/. [C:\projects\junit-testlogger\src\package\package.csproj]
C:\Program Files\dotnet\sdk\3.1.401\Sdks\NuGet.Build.Tasks.Pack\build\NuGet.Build.Tasks.Pack.targets(198,5): error NU5127:  [C:\projects\junit-testlogger\src\package\package.csproj]

@Siphonophora I just remembered we did a similar VS 2019 exercise for NUnit logger, this PR may help: https://github.com/spekt/nunit.testlogger/pull/62

b05a8696e477423e43d453ca1deed8c955289230 Seems to produce the right package. When pushed to a local feed, visual studio sees the icon, but not on myget. So maybe its the compatability of the myget servers? https://github.com/NuGet/Home/issues/8189 suggests thats possible

@codito The previous pull from nunit was very helpful. I think this is all set. As I noted above, I don't see the icon on the feed from myget, but I think it may be lack of support on their end. The package looks to be building correctly now.
If this looks ok, I would squash these commits before merging.


org/arangodb/request: Content-Type: undefined

on arangodb 2.8.4 i noticed that the request module -- as in var request = require('org/arangodb/request'); -- sets the Content-Type: header to "undefined".
I tried to interoperate with resteasy (http://resteasy.jboss.org/) which did not like this header.
According to https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7 the media type must be of form type/subtype.
Thanks

This seems to be a pre-existing problem with how the content-type header is added:
https://github.com/arangodb/arangodb/blob/2.8/js/common/modules/org/arangodb/request.js#L134
Basically the header will always be set even if the value is undefined.
Node's ClientRequest throws on undefined whereas the npm request module silently discards all undefined headers. I think we should do the latter.

Thanks. Much appreciated.


Change default path when opening files

This is an attempt to use the active pane and active item to determine the default path when opening a file (on all platforms).
Note: Not tested yet. See #6762 for the discussion around this bug/feature.

This has been fixed in #11618. Thanks very much for your work on fixing this bug.


Add ability to toggle the menu bar on Windows and Linux

A lot of software on Windows (e.g. Sublime Text, Firefox) allow you to collapse the menu bar so there's only the tab bar. The menu bar can then be collapsed/expanded by pressing the Alt key.
It would be great to have this feature in Atom as it maintains the look and feel across different platforms and gives slightly more vertical space.
(By menu bar I mean the "File / Edit / View / etc" bar.)

Sadly, this does not work on my current (0.177.0) or past linux installs. Strangely enough this worked on windows. Can you help me out? how do I even troubleshoot something like this?

Just to update myself: This is working fine for me on Windows 7 and Fedora 21/GNOME 3. Thanks, @kevinsawicki !
@TheChymera System details for those installs might help. Distro, window manager, desktop environment, etc.

@jahed
gentoo linux, GNOME-3.14 (though I have had this issue since I first installed atom on 3.10).

Do you have the relevant option enabled in settings?

@50Wliu ok, all I needed to do was tick the respective option under Settings. So this works, just that it's not the default behavior. Sorry for the confusion and thanks for the great editor!

When I push tab, it show the bar but does not allow me to interact with it using keyboard. Is this a bug?

@louisgv yes.  See #4057.


[ROCM] Fix export harness tests

@hawkinsp @superbobry could you please take a look?

Indeed, this does fail the export_back_compat_test:
    self.assertEmpty(not_covered,
AssertionError: {'hipsolver_syevd', 'hipblas_geqrf_batched', 'hipsolver_orgqr', 'hip_threefry2x32', 'hipsolver_syevj', 'hip_threefry2x32_ffi', 'hipsolver_geqrf'} has length of 7. : The following custom call targets are declared stable but are not covered by any tests: {'hipsolver_syevd', 'hipblas_geqrf_batched', 'hipsolver_orgqr', 'hip_threefry2x32', 'hipsolver_syevj', 'hip_threefry2x32_ffi', 'hipsolver_geqrf'}

Indeed, this does fail the export_back_compat_test:
    self.assertEmpty(not_covered,
AssertionError: {'hipsolver_syevd', 'hipblas_geqrf_batched', 'hipsolver_orgqr', 'hip_threefry2x32', 'hipsolver_syevj', 'hip_threefry2x32_ffi', 'hipsolver_geqrf'} has length of 7. : The following custom call targets are declared stable but are not covered by any tests: {'hipsolver_syevd', 'hipblas_geqrf_batched', 'hipsolver_orgqr', 'hip_threefry2x32', 'hipsolver_syevj', 'hip_threefry2x32_ffi', 'hipsolver_geqrf'}

It is good to know that the protection works. But the question is if we can just disable that protection for these custom call targets on grounds that if we ever change the hipsolver targets we would change at the same time the cusolver ones, so they should have identical compatibility.

@superbobry @gnecula @hawkinsp Just fixed! Could you please review? thanks!

Thank you! It is almost ready, please see the comment.

Thanks so much! Just fixed!

@Ruturaj4 — It looks like the linter is failing (because of extra newlines in the added files, I think?). Could you take a look and/or run pre-commit locally (as described here). Thanks!

@Ruturaj4 — It looks like the linter is failing (because of extra newlines in the added files, I think?). Could you take a look and/or run pre-commit locally (as described here). Thanks!

@dfm Should be good now! thanks!

@Ruturaj4 — Thank you! Looks good now. Can you squash into one commit then we can merge!

@Ruturaj4 — Thank you! Looks good now. Can you squash into one commit then we can merge!

Done thanks!


webpki-ccadb: update Moz CCADB root report url

Per Chris Clements' update to the mailing list

We recommend users review their linked bookmarks, automation scripts, integration programs, or similar and:
Change their references to public report URLs
from:
ccadb-public.secure.force.com (non-enhanced)
ccadb-public.force.com (non-enhanced)
to:
ccadb.my.salesforce-sites.com (enhanced)

test-only metadata change, going to merge with 1 review.


Update getLargestDisk.ts

adds a log message when the disk it too small, fixes https://github.com/EpicsDAO/solv/issues/90 by giving the users a warning

I am using a 2tb hdd and I'm told hd too small
I'm on GCP - any advice would be appreciated


Customizable Rollups: Count rollups should roll up Opportunities with null Amount field

Rollups with Count operation are not working when a Opportunity Amount is null

Included in beta release 3.130 (Beta 3)

Included in production release 3.130


Change of height/h not picked up

I have a situation where grid components resize themselves based on input, like a form expanding for additional choices or a variable amount of data being loaded and displayed. My code updates the "h" property of the layout object but this does not seem to have the desired effect - The exact behavior is one grid component sitting partially over another, where the grid clearly thinks the component is still the size before it was changed.   Adding a new component or removing an existing one from the grid seems to trigger a fix to the issue.
So in essence, changing the "h" property of the grid layout in-flight does not always immediately take effect.  Is there a work around for this?

Hi @gavbrennan
Can you give a minimal code example? You can use the editable demo as a starting point: https://codesandbox.io/s/5wy3rz5z1x?module=%2Fsrc%2FShowcaseLayout.js

Hi James,
Have updated that demo - please let me know if you see any obvious mistakes!
Best,
Gavin
From: James Ramm<EMAIL_ADDRESS>Sent: 03 March 2021 10:00
To: react-grid-layout/react-grid-layout<EMAIL_ADDRESS>Cc: gavbrennan<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [react-grid-layout/react-grid-layout] Change of height/h not picked up (#1400)
Hi @gavbrennanhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgavbrennan&data=04|01||695ecdd6f58047d6b31a08d8de1a5f94|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637503552139022384|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=xJ8pqsEkoQge7leDu7Paj4eOFz6c4fI4G5CxXlay%2FaY%3D&reserved=0
Can you give a minimal code example? You can use the editable demo as a starting point: https://codesandbox.io/s/5wy3rz5z1x?module=%2Fsrc%2FShowcaseLayout.jshttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcodesandbox.io%2Fs%2F5wy3rz5z1x%3Fmodule%3D%252Fsrc%252FShowcaseLayout.js&data=04|01||695ecdd6f58047d6b31a08d8de1a5f94|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637503552139032381|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=9JCgIs8oQGiBLvjVHfnHJDCOwIPwtFJ59G%2Fe7Re5w9Q%3D&reserved=0

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Freact-grid-layout%2Freact-grid-layout%2Fissues%2F1400%23issuecomment-789519056&data=04|01||695ecdd6f58047d6b31a08d8de1a5f94|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637503552139032381|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=nfpLqjX4EYVLAjJraBkLLrVCoMTd8autECO8R9tviHA%3D&reserved=0, or unsubscribehttps://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADZHRNZFFEH3PXP2JZTXME3TBXUAZANCNFSM4XNDFB2Q&data=04|01||695ecdd6f58047d6b31a08d8de1a5f94|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|637503552139042371|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=K%2F%2F5ydSuzX4hhm2v4%2BvsFHTKQPCAfiOR2SR7pjPcPCA%3D&reserved=0.

I have the exact same problem... @gavbrennan, have you found a solution to your problem?

Not as yet - I was hoping the code example (https://codesandbox.io/s/morning-fog-7oqoq) would get some attention but I guess not.  I've tried to force changing of all component keys on an update of height/width from within a component, but that failed miserably.  So still stuck!

@JamesRamm - did you have chance to a look at https://codesandbox.io/s/morning-fog-7oqoq  ?

@gavbrennan
I'll need to check as I'm in my phone, but at first glance it looks like you are mutating the layouts array in place, which can be a problem for Reacts' diffing algorithm, you really need an immutable update here

Thanks - that solved it.  Specifically, when adjusting the height on a component I created a new instance of the GridLayout.Layout object, copying all other properties across and inserting the new height.  It now behaves as expected, with objects dynamically updating their own height and the grid adjusting accordingly.

hi @gavbrennan I'm also stuck with same issue but unable to figure out how did you create a new instance of GridLayout?
It would be really helpful if you can share some snippet for the same.

@priyankapyt This had me scratching my head a bit too, turns out if you use data-grid it doesn't look like you can easily change the parameters, but if you use the layout prop then it works. Just make sure to always return a new ref.


implement thread::detach()

it's currently not implemented (silently returns) but there is no note in the documentation about it

Right, I remember by the time I wrote it I had no idea what to do.
But the documentation must be updated anyway.
Do you have a use case where you need functionality added to it?

Right, I remember by the time I wrote it I had no idea what to do.
But the documentation must be updated anyway.
Do you have a use case where you need functionality added to it?

Do you have a use case where you need functionality added to it?

When you start a thread as fire-and-forget you don't want to keep a handle to the thread. So you call detach() on it before it goes out of scope.
Another option would be to implement std::async but that forces you to the std api that doesn't allow to set prioriities and stack size.

Do you have a use case where you need functionality added to it?

When you start a thread as fire-and-forget you don't want to keep a handle to the thread. So you call detach() on it before it goes out of scope.
Another option would be to implement std::async but that forces you to the std api that doesn't allow to set prioriities and stack size.

you don't want to keep a handle to the thread

I guess this makes more sense in C with POSIX threads, but in C++, if the thread is a local (stack) object, and you exit the scope, the object is destroyed anyway.
If the thread was allocated somewhere else (dynamically or statically), it is alive as long as that scope is alive, so detach again has not much to do.
Am I missing something?

you don't want to keep a handle to the thread

I guess this makes more sense in C with POSIX threads, but in C++, if the thread is a local (stack) object, and you exit the scope, the object is destroyed anyway.
If the thread was allocated somewhere else (dynamically or statically), it is alive as long as that scope is alive, so detach again has not much to do.
Am I missing something?

I would say it's as if you would "move" the thread into the scheduler:
Once you detach, the thread continue running until it exits. The destructor doesn't have any effect anymore in that situation. Ressources are freed once the thread exits (see cpp reference).

I would say it's as if you would "move" the thread into the scheduler:
Once you detach, the thread continue running until it exits. The destructor doesn't have any effect anymore in that situation. Ressources are freed once the thread exits (see cpp reference).

That's probably possible with std::thread, which is only a wrapper over an internal thread, but os::thread is the actual object, with resources used by the scheduler, it normally cannot survive if moved somewhere else (or things get very complicated).

That's probably possible with std::thread, which is only a wrapper over an internal thread, but os::thread is the actual object, with resources used by the scheduler, it normally cannot survive if moved somewhere else (or things get very complicated).

I don't know how to implement this, but for example CMSIS-RTOS seems to able to do that
(I know it's not C++ but still it might be worth checking the possibility once the threads are revisited)

I don't know how to implement this, but for example CMSIS-RTOS seems to able to do that
(I know it's not C++ but still it might be worth checking the possibility once the threads are revisited)

Ok, will do.

Ok, will do.


Metal fails to initialize with: ggml_metal_init: error: Error Domain=NSCocoaErrorDomain Code=258 "The file name is invalid."

LocalAI version:
LocalAI version v1.20.1-57-gd0e67cc (d0e67cce7550389b657d37bc5956ce4a9e925321)
(compiled with Metal flag)
Environment, CPU architecture, OS, and Version:
Local, Apple M1 Pro, macOS Ventura 13.4.1
Describe the bug
When trying to load a model with the gpu_layers: 1 and f16: true parameters set in the model yaml file, Metal initialization appears to fail. The models run without these parameters set and create an answer.
To Reproduce
Build the project with Metal enabled. Add gpu_layers and f16 parameters to model yaml file according to comment in the documentation for how to integrate Metal support. Send question to LocalAI using the selected model.
Expected behavior
Metal initialization succeeds and LocalAI generates response with Metal acceleration.
Logs
The ggml_metal_init output at the end only appears when the aforementioned parameters are set in the model's yaml file. The ggml_metal_init output is not present when those parameters are not set.
DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:<nil>} sizeCache:0 unknownFields:[] Model:/PATH/TO/LocalAI/models/vicuna-7b-1.1.ggmlv3.q4_0.bin ContextSize:1024 Seed:0 NBatch:512 F16Memory:true MLock:false MMap:false VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:1 MainGPU: TensorSplit: Threads:4 LibrarySearchPath:}
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr llama.cpp: loading model from /PATH/TO/LocalAI/models/vicuna-7b-1.1.ggmlv3.q4_0.bin
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr llama_model_load_internal: format     = ggjt v3 (latest)
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr llama_model_load_internal: n_vocab    = 32000
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr llama_model_load_internal: n_ctx      = 1024
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr llama_model_load_internal: n_embd     = 4096
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr llama_model_load_internal: n_mult     = 256
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr llama_model_load_internal: n_head     = 32
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr llama_model_load_internal: n_layer    = 32
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr llama_model_load_internal: n_rot      = 128
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr llama_model_load_internal: ftype      = 2 (mostly Q4_0)
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr llama_model_load_internal: n_ff       = 11008
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr llama_model_load_internal: model size = 7B
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr llama_model_load_internal: ggml ctx size = 3615.72 MB
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr llama_model_load_internal: mem required  = 5407.72 MB (+ 1026.00 MB per state)
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr llama_new_context_with_model: kv self size  =  256.00 MB

>>>> RELEVANT PORTION <<<<<<
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr ggml_metal_init: allocating
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr ggml_metal_init: using MPS
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr ggml_metal_init: loading '(null)'
DBG GRPC(llama-vicuna-7b-1.1.ggmlv3.q4_0.bin-localhost:63879): stderr ggml_metal_init: error: Error Domain=NSCocoaErrorDomain Code=258 "The file name is invalid."
>>>> RELEVANT PORTION <<<<<<

Additional Information
Tried with the following models:
vicuna-7b-1.1.ggmlv3.q4_0.bin
nous-hermes-13b.ggmlv3.q4_0.bin
Wizard-Vicuna-13B-Uncensored.ggmlv3.q4_0.bin

I can run go-llama / llama.cpp from within the go-llama folder with Metal acceleration. I've tried copying the ggml-metal.metal into different folders that would make sense to load it from but with no success.
I would expect it to be loaded from the CWD that I have LocalAI installed to as the Makefile indicates

hey @philippjbauer , I don't have an Apple HW, could you give a shot at #777?

This worked for me: just copy ggml-metal.metal to wherever you copied the executable (or libllama.so, I guess) to.
For example, if you cp ./main /tmp/main, then you also have to cp ./ggml-metal.metal /tmp, otherwise /tmp/main won't run when MPS is enabled.
The problem is in ggml-meta.m, where path becomes null when ggml-meta.metal isn't in the same directory as the executable:
    // read the source from "ggml-metal.metal" into a string and use newLibraryWithSource
    {
        // ...
        NSBundle * bundle = [NSBundle bundleForClass:[GGMLMetalClass class]];
        NSString * path = [bundle pathForResource:@"ggml-metal" ofType:@"metal"];
        fprintf(stderr, "%s: loading '%s'\n", __func__, [path UTF8String]);

        NSString * src  = [NSString stringWithContentsOfFile:path encoding:NSUTF8StringEncoding error:&error];
        if (error) {
            // this error got triggered
            fprintf(stderr, "%s: error: %s\n", __func__, [[error description] UTF8String]);
            exit(1);
        }

after building the ggml-metal.m file should be already next to the binary - at least after #777 . can you confirm ?

Yes, you are right, mudler!
By default, ggml-metal.metal (not ggml-metal.m, which gets compiled and linked into the executables) is already in the same directory as the executables, and nothing is required to run the models using MPS.
I posted that comment because I happened to move ./main around and the exact same error came up, and I was able to resolve it by moving ggml-metal.metal to the same directory.
Maybe philippjbauer ran into a different problem than mine, but the error message is the same as mine, and I searched through the entire code base, there's only one place for that message to happen.


Version 4 alpha 6: .d-flex and .row doesn't work in Internet Explorer

Search for duplicate or closed issues
Nothing found: https://github.com/twbs/bootstrap/issues?utf8=✓&q=is%3Aissue is%3Aopen .d-flex internet explorer

Operating system and version (Windows, Mac OS X, Android, iOS, Win10 Mobile)
Windows 8.1

Browser and version (Chrome, Firefox, Safari, IE, MS Edge, Opera 15+, Android Browser)
Internet Explorer 11.0.xxx and Chrome Version 58.0.3029.110 (64-bit)

Reduced test cases and potential fixes using JS Bin:
This works in Chrome and iOS Safari but not Internet Explorer.
https://jsfiddle.net/f985yydv/1/
It should center the row on the page. In Internet Explorer, when the page is wider than 'medium', it shrinks the row to only about 10% of the width for no reason. If you remove 'd-flex' class then it appears correctly, but the row is not vertically centered.

Example Headline

        Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.

This might be the same issue as #21590. Can you test with the latest from v4-dev branch?

Hey there!
We're automatically closing this issue since the original poster (or another commenter) hasn't yet responded to the question or request made to them 14 days ago. We therefore assume that the user has lost interest or resolved the problem on their own. Closed issues that remain inactive for a long period may get automatically locked.
Don't worry though; if this is in error, let us know with a comment and we'll be happy to reopen the issue.
Thanks!
(Please note that this is an automated comment.)


Deploy to vercel

Great project! How can I deploy this repository to Vercel, and what settings should be added?

Hi, I'm not sure why you want to deploy to Vercel, because unlike many Github blog repos, this one doesn't need to be deployed to be used: you can check the how-it-works  section.
If you're certain you want to deploy Tinymind yourself, you need to create a .env file with the following content:
GITHUB_ID= # your Github app id
GITHUB_SECRET= # your Github app secret
NEXTAUTH_SECRET= # random generate one
NEXTAUTH_URL=http:localhost:3000

Then you need to create an application in Github and paste the GITHUB_ID and GITHUB_SECRET. Then run:
npm i
npm run dev