[API] Function count different in GhidraScript vs not w/ headless API

Describe the bug
I'm running Ghidra headless using its API and I've got some code that does some post-processing on the Program object after it is analyzed with Ghidra's headless analyzer. I've noticed that if I try to get the functions for the program within the context of a GhidraScript vs within just the same context as the headless analyzer, the number of functions yielded via program.getListing().getFunctions(true) differs.
E.g. For the "Paranoid Fish" (https://github.com/a0rtega/pafish) program, I'm getting 147 total functions when running my postprocessor code in a GhidraScript. When I run the same postprocessor code not as a GhidraScript, I'm getting 145 total functions.
To Reproduce

Using Ghidra's API from the ghidra.jar file produced by the buildGhidraJar utility, analyze a program with the Ghidra headless analyzer API: AutoAnalysisManager.getAnalysisManager(myProgram).startAnalysis(myMonitor)
After call to startAnalysis completes, use myProgram.getListing().getFunctions(true) and count the number of functions.
Execute a GhidraScript that uses myProgram.getListing().getFunctions(true) and counts the number of functions in there.
Observe a difference in the count of functions depending on the context in which the postprocessor code runs.

Expected behavior
Function count in both versions of the post processor code (GhidraScript vs not in GhidraScript) is expected to be the same.
Environment (please complete the following information):

OS: Ubuntu 22.04.5 LTS
Java Version: 21.0.4
Ghidra Version: 11.2.1
Ghidra Origin: official GitHub distro

Here is the full (minimal) post-processor code I'm trying to run which I'm experiencing this discrepancy with:
    public void resolveUndefinedFunctions(Program program, TaskMonitor monitor)
            throws CancelledException {
        // Mostly just snarfed from an existing Ghidra script.
        AddressSet orphanSet = new AddressSet();
        Listing listing = program.getListing();

        InstructionIterator initer = listing.getInstructions(program.getMemory(), true);
        while (initer.hasNext() && !monitor.isCancelled()) {
            Instruction instruct = initer.next();
            orphanSet.addRange(instruct.getMinAddress(), instruct.getMaxAddress());
        }
        FunctionIterator iter = listing.getFunctions(true);
        while (iter.hasNext() && !monitor.isCancelled()) {
            Function f = iter.next();
            orphanSet.delete(f.getBody());
        }

        logger.info("resolveUndefinedFunctions orphan set size: %s".formatted(orphanSet.getNumAddressRanges()));

        if (orphanSet.getNumAddressRanges() == 0) {
            return;
        }

        int newFuncCount = 0;
        int blockCount = 0;

        IsolatedEntrySubModel submodel = new IsolatedEntrySubModel(program);
        CodeBlockIterator subIter = submodel.getCodeBlocksContaining(orphanSet, monitor);
        while (subIter.hasNext()) {
            CodeBlock block = subIter.next();
            blockCount++;
            Address deadStart = block.getFirstStartAddress();
            Function func = listing.getFunctionContaining(deadStart);
            if (func != null) {
                orphanSet.delete(deadStart,deadStart);
                continue;
            }

            Instruction instr = listing.getInstructionAt(deadStart);

            while (instr != null) {
                Function fun = listing.getFunctionContaining(instr.getAddress());
                if (fun != null) {
                    break;
                }
                boolean funCreated = createFunctionAt(program, instr.getAddress());
                if (funCreated) {
                    printf("Function created at %s.\n",instr.getAddress());
                    newFuncCount++;
                }
                instr = instr.getNext();
            }

        }
        logger.info("Count of resolved undefined functions: %s".formatted(newFuncCount));

        FunctionIterator fiter = program.getListing().getFunctions(true);
        int fcount = 0;
        for (Function func : fiter) {
            fcount++;
        }
        logger.info("Total function count: %s".formatted(fcount));
    }

    protected boolean createFunctionAt(Program program, Address entry) {
        Function func = program.getFunctionManager().getFunctionAt(entry);
        if (func == null) {
            CreateFunctionCmd createFunctionCmd = new CreateFunctionCmd(entry);
            createFunctionCmd.applyTo(program);
            func = program.getFunctionManager().getFunctionAt(entry);
            if (func == null) {
                return false;
            }
            return true;
        }
        return false;
    }

Similar to #2639 and #5531

Updated to include minimal sample code exhibiting the discrepancy.

I think there is some misunderstanding. I'm not running the headless analyzer with the analyzeHeadless utility packaged with Ghidra. I built the ghidra.jar file using the buildGhidraJar utility, and I'm using that in a custom Java application.

Yes, there was as misunderstanding.  I was confused by your use of the term postScript, which only has meaning to me in the context of a HeadlessAnalyzer Java object.
I now get that you are using a ghidra.jar as a library being used by some other program, but why are not going through the HeadlessAnalyzer class API?  This is the same API that calling analyzeHeadless from the command line uses.
Your use case would make more sense to me if i could see all of your code.  For example, I don't understand how you are getting myProgram.

I'm not going through the HeadlessAnalyzer class API because I need access to the Program object after the analysis for my postprocessing and output needs (not as a postscript).
This is how the application is getting myProgram:
public Program loadProgram(File exeFile, TaskMonitor monitor) throws VersionException, InvalidNameException, DuplicateNameException, CancelledException, IOException {
	ProjectManager projectManager = new MyProjectManager() {};
	ProjectLocator locator = new ProjectLocator("absolute/path/to/projects/", "MyProject");

	Project project = new DefaultProject(projectManager , locator, false);
	MessageLog log = new MessageLog();

	LoadResults<Program> progLoadResults = AutoImporter.importByUsingBestGuess(exeFile, project , null, this, log, monitor);
	return progLoadResults.getPrimaryDomainObject();
}

// This exists just to allow access to the constructor
private static class MyProjectManager extends DefaultProjectManager {}

After the HeadlessAnalyzer is finished, you should be able to open up the program from the project with something like
DomainFile domainFile = project.getProjectData().getFile("/path/to/program/in/project");
Program program = (Program) df.getDomainObject(consumer, true, true, monitor);

ok. I'll try redesigning my application to use that instead. Thanks!

I'm having some issues with DomainFile df = project.getProjectData().getFile(path/to/my/sample.exe). When I call this with the path to the exe I'm trying to analyze, the domainFile I get back is null. Is there some additional setup I need to do?

The project is a temporary project that I create just before doing the headless analysis. It is then deleted afterwards. Should I try copying my program into the project's folder before analyzing it and get the DomainFile for that?

I just wanted you to make sure your path string was right.  I forget if it should start with a /.  You could also try looping through as a sanity check with something like projectData.getRootFolder().getFiles().  These are all just debugging suggestions.

Okay, I finally got that to get the DomainFile correctly. I had to copy my exe into the project's folder, then construct a path that looks like /myProgram.exe corresponding to the actual path rootFolderPath/projectName/myProgram.exe.
It seems that in the results, I'm still seeing the original issue where it is missing the couple of functions though. :/

Can you share how you calling the HeadlessAnalyzer class?

	public void processLocal(String projectLocation, String projectName, String rootFolderPath,
			File file, ProgramHandler programHandler, TaskMonitor monitor) throws IOException {

		// If we are importing, need some files to import or at least a script to run!
		if ((filesToImport == null || filesToImport.size() == 0)) {
			Msg.warn(this, "REPORT: Nothing to do ... must specify file(s) for import.");
			return;
		}

		// // If importing, add trailing slash if it isn't there so that non-existent folders are created
		if (!rootFolderPath.endsWith("/")) {
			rootFolderPath += "/";
		}

		// Open the Ghidra project.
		File dir = new File(projectLocation);
		ProjectLocator locator = new ProjectLocator(dir.getAbsolutePath(), projectName);
		if (locator.getProjectDir().exists()) {
			project = openProject(locator);
		}
		else {
			Msg.info(this, "Creating " + (options.deleteProject ? "temporary " : "") +
				"project: " + locator);
			project = getProjectManager().createProject(locator, null, false);
		}
		project.close();

		// Analyze each file in `filesToImport` using HeadlessAnalyzer.
		try {
			Msg.info(this, "Doing headless analysis on sample %s".formatted(file.getAbsolutePath()));

			// Copy the sample into the project's folder.
			File projectFile = new File(locator.getProjectDir().getPath() + "/" + file.getName());
			Files.copy(Paths.get(file.getPath()), Paths.get(projectFile.getPath()));

			List<File> singleFileList= new ArrayList<>();
			singletonList.add(file);

			Program program = null;
			try {
				// Perform the headless analysis.
				HeadlessAnalyzer analyzer = HeadlessAnalyzer.getInstance();
				analyzer.processLocal(projectLocation, projectName, rootFolderPath, singleFileList);

				// Need to get the path of the sample relative to the rootFolderPath.
				String filePathInProject = "/" + file.getName();
				Msg.info(this, "Path to exe in project: %s".formatted(filePathInProject));

				// Access the program and run the postprocessing hook on it.
				project = openProject(locator);
				DomainFile df = project.getProjectData().getFile(filePathInProject);
				program = (Program) df.getDomainObject(programHandler, true, true, monitor);
				programHandler.postProcessHandler(program);
			}
			catch (Exception e) {
				String logErrorMsg = file.getAbsolutePath() + " Error during analysis: " + e.getMessage();
				Msg.error(this, logErrorMsg);
				e.printStackTrace();
			}
			finally {
				program.release(programHandler);
			}
		}
		finally {
			// Delete the temporary Ghidra projects folder.
			project.close();
			FileUtilities.deleteDir(locator.getProjectDir());
			locator.getMarkerFile().delete();
		}
	}

	private Project openProject(ProjectLocator locator) throws IOException {
		Project tempProject;

		if (options.deleteProject) {
			Msg.warn(this, "Project already exists and will not be deleted: " + locator);
			options.deleteProject = false;
		}

		Msg.info(this, "Opening existing project: " + locator);
		try {
			tempProject = new HeadlessProject(getProjectManager(), locator);
		}
		catch (NotOwnerException e) {
			log.error("Failed to open project due to {}", e.getMessage(), e);
			throw new IOException(e);
		}
		catch (LockException e) {
			log.debug("Failed to open project due to {}", e.getMessage(), e);
			throw new IOException(e);
		}

		return tempProject;
	}

	private HeadlessGhidraProjectManager getProjectManager() {
		if (projectManager == null) {
			projectManager = new HeadlessGhidraProjectManager();
		}
		return projectManager;
	}

	/**
	 * Ghidra project class required to gain access to specialized project constructor
	 * for URL connection.
	 */
	private static class HeadlessProject extends DefaultProject {
		HeadlessProject(HeadlessGhidraProjectManager projectManager, ProjectLocator projectLocator)
				throws NotOwnerException, LockException, IOException {
			super(projectManager, projectLocator, false);
		}
	}

	private static class HeadlessGhidraProjectManager extends DefaultProjectManager {
		// this exists just to allow access to the constructor
	}

What is this doing?
programHandler.postProcessHandler(program);

I was expecting you to use analyzer.getOptions().setPostScripts() to execute your GhidraScripts.

The goal in suggesting the use of the HeadlessAnalyzer class was to mimic what happens when you run from the command line.

That bit is running the code from the original post that I was originally running as a post-processor script. Then I was tasked with refactoring things so that the same post-processor code runs in the same Java context as the rest of the application. That's when I discovered the discrepancy in the function count.

What do you mean "in the same Java context?"  GhidraScripts are run in the same JVM as the Ghidra application.

I mean like the GhidraScripts don't have access to modules from the rest of my application, only those present in the GhidraScripts path, and they appeared to me to run in a separate JVM. I observed this like a few versions (about a year) ago. Maybe it's not like that now? Or maybe my observations were wrong. I don't know.

Okay, I see that you're right now about GhidraScripts running in the same JVM.

Is there a way to access modules in elsewhere in my application from within the GhidraScript? If that's possible, I think I could work with my postprocessing code running as a GhidraScript.

I'm thinking the problem might have something to do with how GhidraScripts now use OSGi.  Tomorrow i'll take a look at that.  I think we are explicitly specifying what packages get imported into the GhidraScript and it might just be Ghidra-related ones.

What error do you get at runtime when it can't find your application's classes?

Nevermind, it seems that I am able to call modules in my application's jar from a GhidraScript after all. :)


Switch to shell script from Gradle runner at bitrise

What has been done
Temporarily disabled Gradle runner step because of https://github.com/vgaidarji/ci-matters/issues/35.
Fixes #27
Alternative solution
Just use custom version of gradle runner step as follows (from this commit)
    - git::https://github.com/bitrise-io/steps-gradle-runner.git@unset_ndk_home:
        title: Gradle Build/Test/Analyse/Distribute
        inputs:
        - gradle_task: assembleDebug testDebug jacocoTestReport checkstyle pmd jdepend
            lintDebug buildDashboard assembleDebugAndroidTest crashlyticsUploadDistributionDebug
            -PversionCode=$BITRISE_BUILD_NUMBER -PfabricApiKey=$FABRIC_API_KEY -PfabricApiSecret=$FABRIC_API_SECRET
        - mapping_file_exclude_filter: ''

Coverage remained the same at 100.0% when pulling 4a4d00df5a1f25275f3b60e42cd923419297c0e2 on fix-bitrise-build into c421f997c1b3c3436325783a08b02d2984b6cd70 on master.


Optimize puzzle lookups

I think it should be possible to optimize puzzle lookups by doing them in parallel, even if you're requesting from the same peer. There should be a rate limiter in place, but we can still do them in batches to prevent overloading peers with requests or blocking the rest of the wallet.

This is fixed in https://github.com/xch-dev/sage/pull/172, it will now sync 5 from the same peer at the same time


missing keytool?

Issued the certificate perfectly fine, but installing doesn't work... it's missing the keytool binary? I wasn't able to find it anywhere on the machine?! did they remove java on it? I looked for openjk and apt is only showing release in an "oldstable" source... not sure if I'm supposed to install that!?
 Working fine :)

@mason-xx glad to her it works!!


cleanup ci scripts, badges, and unrelated and unwanted install script

most of the badges were already taken care of, there really wasn't much to clean up.

Codecov Report

Merging #68 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop      #68   +/-   ##
========================================
  Coverage    83.07%   83.07%
========================================
  Files           11       11
  Lines         1176     1176
========================================
  Hits           977      977
  Misses         199      199

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1563a9b...8477e87. Read the comment docs.


#332: Added default fallback language to English

#332 Default fallback language was never set and setting a default language did not request for the translations file if the language was not recognised.
Also added a version to the package.json since:

The most important things in your package.json are the name and version fields. Those are actually required, and your package won't install without them.

Source: https://docs.npmjs.com/files/package.json#version
Without this version I could not reference to my fork version on GitHub from my own project's package.json file. If (for some reason) you do not want it, you may remove it of course.
Let me know what you think!

The version is automatically generated on Travis when I push / merge commits (based on the commits messages), that's why it's empty in package.json :)

@ocombe Ahhh right, my bad! Shall I revert the commit?

Just remove it and push the changes. We can squash commits here.

Done. :)

So my point of opinion on this PR:

We should not set the default language to "en" automatically, we just don't know how people want to name it, and maybe for some reason they don't want a default language
Is it really necessary to have a DefaultLangChangeEvent ? I don't see any use case, you define the default lang at bootstrap and then you forget about it, it's not really something that you should change
If we don't use the DefaultLangChangeEvent, then changeDefaultLang becomes unecessary as well

So in the end, the only thing that you should do in this PR is: get the default lang file when the current translation is missing (if the default lang is missing too)

And sorry if I seem a bit blunt here, I'm just making sure that we don't bloat the code too much by adding stuff that isn't necessary. But rest assured that I appreciate the time that you took to do this PR, and it's a good one (clean code and tests!)

@ocombe Thank you for your feedback! No worries, this is the kind of feedback I can use. We all want the same thing, to make it as good as we can! :)
To get back on your 3 points of attention:

You are completely right, didn't think of that. Will update the PR!
2 & 3) I see what you mean and I tried to do it without the default lang changed event. The problem is though that I need to inform the component to update all keys when the default language changes (like with a normal use updates all translations as well). I could "abuse" the event that is already there for the use to update all translations. But this could lead to some strange behaviour (e.g. waiting for a use to finish, but getting a default lang changed event in between).

Removed the default language to be 'en'. Moved the logics that I put into setDefaultLang (which is not really functionality for a setter function) to the getParsedResult function. Also updated the unit tests to reflect the changes.

Ok good, a few last changes and I'll merge it:

fix the conflicts (merge master into your branch)
update the new directive with the same thing that you did in pipe
add documentation on the new event in the README.md

Conflicts are merged
Updated new directive and added some extra unit tests (also for the new subscription). This made me realise that moving the get default language translations from setDefaultLang to getParsedResult was wrong. Setting the default language should emit an event with the translations for the default language. In theory the setDefaultLang function should work about the same way as the use function. So I moved it back to setDefaultLang (but left out the part from getParsedResult where it set the default lang to 'en' of course).
Added documentation about the new event in the README.md

Thanks for your help!!

@ocombe Updated PR, removed the translations argument from the DefaultLangChange events in the pipe and the directive.

It's ready to be merged. Let me know if I am missing something!

@ocombe Can this be merged?

@ocombe Is this on hold/waiting for something?

Sorry, been a bit busy, I'll try to do it this week

No worries, not trying to insult or push you. I was just wondering what the status was.

No problem :)

Ok I fixed the bug in the directive, the failing test will pass now.
I also fixed a few other things at the same time: it sets the default lang to the first lang that receives translations (if the default lang is undefined at that time), and the onTranslationChange event now emits for all setTranslation calls (not just for merges, I don't know why this was just for merge).
Also the directive now listens to onTranslationChange events.

All of this is available in the new 5.0.0 release.

Nice, great work!


[Question] What's the point of CPUFriend? (& other things)

Hey! First of all, the work you've been doing for BigSurface and for various Hackintosh projects in general is very admirable. Thanks for that!
(Note: this issue is really long. I have an unfortunate habit of making GitHub issues/PRs far too long, and this issue unfortunately follows that pattern, covering far too much for one issue.)
Anyway, I've been using an SL3 (i7, 16GB RAM, 13-inch) Hackintosh as one of my daily drivers for about 7 months now and have never used CPUFriend (I know almost nothing about it, I've only briefly peeked at the code). The i7-1065G7 in the SL3 is, as I'm sure you know, not natively supported by macOS. (The equivalent SMBIOS, MacBookAir9,1, which I use on my SL3, has a i7-1060NG7.) I've heard that CPUFriend is designed to help CPU power management for Hackintoshes that aren't natively supported, so what exactly would it do if using it with SL3?
I ask because I've had two frustrating issues with my SL3 Hackintosh that are almost certainly related to the CPU. First of all, the CPU gets really warm (70c-90c degrees, which is extremely abnormal and unhealthy (measured via Stats) and the entire computer throttles like crazy. It becomes completely unusable, with the screen becoming almost unresponsive. This is natural due to the abnormally high temperature, but do you think CPUFriend could help against overheating? (I usually run into this issue when using the laptop on something like a couch; additionally, the last time I've cleaned the fans was probably over a year ago, so I'll definitely try clean the laptop when I have time.) Second of all, when the computer is below ~10% battery, its becomes exactly like what I previously described — macOS (particularly animations) stutters and takes forever to do simple tasks. I've tested this on various occasions, and it's always the same. Most recently, I ran the battery down to 10% and then shut off the computer. A day later, I shut on the computer again and noticed the same struggles (even bootup was slow, and text in verbose was moving ridiculously slowly). Do note that this issue only happens if the computer is turned on while below 10%. When the laptop is above 10% and goes below 10% after use, performance is always solid like normal (even at 1%). Also, my battery capacity is ~83%. Battery doesn't help long, and I'm hoping CPUFriend could fix that.
I plan on testing some other things with the SL3 soon — I'm coming back from hiatus in the project and will update BigSurface 6.5 to your modified version among other things. Is there any advantage to using the MacBookPro16,2 SMBIOS instead of the MacBookAir9,1 SMBIOS? I know the 16,2 has a i7-1068NG7 — what difference does SMBIOS make performance-wise? Also, completely unrelated, but what HiDPI resolution do you use? I'm rather unhappy with mine, but it's the best BetterDisplay will let me do.
Last, BigSurface is currently behind on various things — dependencies need to be updated and a new release hasn't been published in a while. I assume @Xiashangning is taking a break for now and won't update the kext for a while, so do you plan to take over the kext while they are away?
Thank you in advance for your help, and for reading my long GH issue! I'm happy to help test anything or do anything SL3-related in the future and am grateful that you're providing useful resources for the Hackintosh community. Hopefully you can help me out as an experienced Hackintosher.

I'm closing this issue for a couple of reasons:

It's far too long
I've fixed many of my issues in the above issue and made significant progress in this project
I'm creating another issue that's more up-to-date

Hey ConwayTech-Dev,
Sorry for answering only today, I'm extremely busy at work right now.
Your post isn't too long at all :-)

I've fixed many of my issues in the above issue and made significant progress in this project

Please contribute back to this repo if you fixed important things so everyone could benefit from your findings, your pull requests are welcome!
But just to quickly add my 2 cents:
Your CPU shouldn't be reaching high temperatures like that, my i5 13" and i7 15" SL3s don't heat up like crazy even when I compile stuff with Xcode... Sometimes I'll hear the fans spin up for a few minutes, but not while doing trivial stuff like Firefox / Mail / VLC / Plex.
Regarding the SMBIOS, from what I understand, MacBook Pros usually came with more powerful CPUs than MacBook Airs. The default values from the CPUFriendFriend CLI suggest that the frequency vectors used by default by Apple for the MacBook Airs are a bit more oriented towards performance to compensate for the weaker CPUs and the MacBook Pros are a tad more oriented towards power saving to increase the runtime on battery.
Changing the SMBIOS should not make a big difference in temperature or performance (if any at all), but using CPUFriend and the CPUFriendFriend tool allows you to define your own frequency vectors for the CPU and thus you could achieve lower temperatures at the expense of lower performance as well. Feel free to play around with CPUFriendFriend, I've tried the lower or lowest settings everywhere once and the OS still runs quite OK, but Apps take ages to start :-)
I personally use the default vectors from my SMBIOS (CPUFriend disabled in my config.plist), which works well for me, providing me with a cool CPU and balanced performance with around 5 hours runtime on battery (i7 SL3 15"). I also set "Low Power Mode" to "Always" in the System Settings, because I don't need much more performance than that.

Also, completely unrelated, but what HiDPI resolution do you use? I'm rather unhappy with mine, but it's the best BetterDisplay will let me do.

I don't have the machine on hand here, but BetterDisplay should give you literally dozens of resolutions for your SL3. There's also a few options to check in the settings so that "better" resolutions are added to the native macOS Display Settings: in "Displays" select your "Built-In Display", then under "System Configuration and Flexible Scaling" enable "Edit the system configuration of this display model". Then restart and you should have nice defaults directly in macOS's Display Settings.

I assume @Xiashangning is taking a break for now and won't update the kext for a while, so do you plan to take over the kext while they are away?

Xiashangning is still active, but very infrequently.
As I don't have the fantastic programming skills he has (I'm no programmer at all, but I understand code quite well as I used to write code quite a lot in another life a very long time ago), I can't really maintain the BigSurface project other than answering a few questions and fixing trivial bugs. So don't expect much from Xiashangning or myself, but I'll try my best to make sure that IceLake Surface devices will still work in macOS Sequoia (or whatever version will be the last one supporting Intel CPUs) with the functionality they have now such as working battery readings, native trackpad gestures and hibernation. Don't expect any new stuff like double battery support or keyboard and trackpad support after detaching and reattaching the clipboard (keyboard part) of the SB3 line, for instance.


Fill empty config label by the label in default language

Questions
Answers

Description?
Fill empty config label by the label in default language

Type?
bug fix

BC breaks?
no

Deprecations?
no

Fixed ticket?
Fixes PrestaShop/PrestaShop#27474

How to test?
Cf. issue

Thanks @marwachelly, it is the wanted behavior. When a field is mandatory, we use the default language as fallback.

Thanks @marwachelly, it is the wanted behavior. When a field is mandatory, we use the default language as fallback.

Thank you @MatShir. So the PR is Ok. :+1:

Thanks @marwachelly


Introduce cluster instance identity other than IP addresses

Scenario: Two Alertmanager clusters (A, B) are running in the same Kubernetes clusters.
If Alertmanager cluster A is scaled down by one instance with IP address X and within a small time range Alertmanager cluster B is scaled up by one instance with the recycled IP address X the two clusters will merge resulting in one big cluster.
We are hitting this problem in the Prometheus Operator end-to-end test suite (https://github.com/coreos/prometheus-operator/pull/2544/) on a single node (small CIDR space) Kubernetes cluster.
The probability of this happening in production systems is questionable. Most setups probably only include a single Alertmanager cluster, in addition, CIDR ranges might be a lot bigger and IP address recycling might not happen as frequently.
This could be prevented via a unique identifier per Alertmanager cluster, disallowing instances with different identifiers to join. In addition https://github.com/prometheus/alertmanager/pull/1819 introducing mutual TLS could stop accidental cluster merges in case trust chains a scoped per cluster.
The purpose of this issue is to document the failure for the future and give anyone hitting the same issue a central place to discuss further precedence.

While TLS chain of trust could accidentally solve this, I don’t think this is the appropriate solution. As you proposed a separate mechanism sounds reasonable.
As for the probability, this is actually not all that low I recall Kubernetes IP recycling to have caused various problems across the board.

Closed by #3354


merge to main branch

merge to main branch

add rocketmq-streams-channel-rocketmq, rocketmq-streams-serviceloader, rocketmq-streams-window modules


[WEB] Add Analytics for User Onboarding Funnel & Bet Intent

All looks reasonable but I defer to others as to how much tracking is appropriate in where.


Unpack hooks now run prior to the package being made immutable.

To facilitate this, archive.Extract() now returns a finalisation
function that makes the package directory immutable. The hooks run in
between extraction and finalisation.
Fixes #1

Looks good, though I think it would be less fragile if the unpack triggers would be executed inside Extract instead of relying in the finalise being called after.

I considered this but it would expand responsibility of archive.Extract() and split responsibility of running hooks over two packages.


merge: stage to master

:tada: This PR is included in version 1.10.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


fix (goals): video URL

Issue 91 https://github.com/OfferZen-Community/developer-mentoring/issues/91
Copied the styling from the commented out portion . Happy to change the colours as needed

@thaliapill Thanks for this! Just checking out what this looks like in light mode vs dark mode — is it possible to have the colours all at #2F96F4 but also underlined (pretty much how it looks when it says "personal SWOT analysis")?

Is this better @lasse-offerzen ? I added in underlining and am using the default colour. If happy, will push the change


amcl doesnot avoid obstacles actively

Hi,
Now amcl will not avoid obstacles if there are new obstacle in the workspace that does not log in gmapping maps.
How should we modify this pkg ?

@hoppss can you give more details about what you are using?  (ie real vs sim)  Also, are you using indigo or kinetic?  I see to explicit issues with this on the navigation repo.

Hi ,
sorry.
We bought both  husky and jackal . I found jackal ( not husky, sorry) didnot avoid obstacles when i run jackal repo 's  navigation package  with a map that i made by gmapping.
It looks like the local planner doesnot work.

@hoppss are the laser's frame and topic correct?

@tonybaltovski you are right!
Thank you very much. May you have a good day.

@hoppss was this integrated by us?

no.
It is well to simulation with jackal gazebo model.
I clone it to my pc to run it with a real jackal, I didnot modify the frame parameters completely.

@hoppss Glad to hear everything is working.  Closing the ticket


On the Site Setting the system redirects to non-existing page after uploading a valid favicon file and pressing an "Update" button

Steps to reprtoduce:

Log in to the site as an Admin
Go to Site Settings
Upload a valid favicon file (you can take this)
Press the "Update" button

Actual Result:
Redirect to non-existing page
Expected Result:
Message in the header "Setting was successfully updated" on the Site Settings page without any redirections

Message in the header "Settings were successfully updated" on the Site Settings page is shown as intended, but only locally.
Suggestion: Possible issues with the Render hosting platform?

Bug was fixed.


Version 1.0.7

This version is not installed 'App not compatible with this device'
Samsung A33
Android 14

Yes, because PPPPS target sdk is 23.
How to install it:
https://github.com/henrichg/PPPPutSettings/blob/main/docs/install_apk_from_pc.md
Use adb command:

for Windows: adb install --bypass-low-target-sdk-block [patch to apk
in PC]\PPPPutSettings.apk
for Linux, OSX: adb install --bypass-low-target-sdk-block [patch to
apk in PC]/PPPPutSettings.apk

st 14. 2. 2024 o 12:24 Orlproi @.***> napísal(a):

This version is not installed 'App not compatible with this device'
Samsung A33
Android 14
—
Reply to this email directly, view it on GitHub
https://github.com/henrichg/PPPPutSettings/issues/5, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA63YWV37ALTY6Z3BPEAL5TYTSNIVAVCNFSM6AAAAABDIDNRICVHI2DSMVQWIX3LMV43ASLTON2WKOZSGEZTIMJTG42DANI
.

GitHub:  https://github.com/henrichg

OK with ADB installed. Thank you


fixed the documentation of glorot_normal according to issue 25564

PR for issue #25564. Modified the documentation of tf.keras.initializers.glorot_normal to specify that the 'stddev' argument refers to the standard deviation of the truncated normal distribution and not the standard deviation of the underling normal distribution.

I think we should then update also the doc for the others. If you agree, I can submit a PR for that.

I think any contribution is welcome. Let's be careful, and only fix the documents which look misleading if we don't correct them :-)

I got an e-mail that said "Closed #25564." Is the issue fixed now? I don't see any change yet from the web https://www.tensorflow.org/api_docs/python/tf/keras/initializers/glorot_normal

Hi, we close the issue because its fix has been merged into master branch. And the web page will be updated after next release version (perhaps tf 2.0).

@facaiy Well done. Now, I understand what is the GitHub for.


I want open imageURL

Image does not open when imageurl is entered in the image option.
Is it not possible to open it?

Image does not open when imageurl is entered in the image option.
Is it not possible to open it?

hi, no, you must write image bytes to the file, and then use the file path for the image option

Image does not open when imageurl is entered in the image option.
Is it not possible to open it?

hi, no, you must write image bytes to the file, and then use the file path for the image option

It seems to be passing the xml full text as a template, but it seems to be a problem in parsing the xml full text. Is there any way to write the full text of xml as it is?

Image does not open when imageurl is entered in the image option.
Is it not possible to open it?

hi, no, you must write image bytes to the file, and then use the file path for the image option

It seems to be passing the xml full text as a template, but it seems to be a problem in parsing the xml full text. Is there any way to write the full text of xml as it is?

https://github.com/MohsenHaydari/flutter_windows_notification/blob/main/lib/windows_notification_method_channel.dart
if you take a look at these codes, I use some pattern like #1# to replace what I want, when I wrote this package I checked many xml like that and I don't see any problem, I suggest you use some XML editor like the android studio to create XML, then add your pattern to XML and when you want to post notification replace the pattern with anything you want like(big text,text,image,type and etc...)

if you take a look at these codes, I use some pattern like #1# to replace what I want, when I wrote this package I checked many xml like that and I don't see any problem, I suggest you use some XML editor like the android studio to create XML, then add your pattern to XML and when you want to post notification replace the pattern with anything you want like(big text,text,image,type and etc...)

Then, if there is no problem, putting the imageurl in the template xml should work, isn't it normal?

I just tested with local image path and i don't know about http, or https url

if you take a look at these codes, I use some pattern like #1# to replace what I want, when I wrote this package I checked many xml like that and I don't see any problem, I suggest you use some XML editor like the android studio to create XML, then add your pattern to XML and when you want to post notification replace the pattern with anything you want like(big text,text,image,type and etc...)

Then, if there is no problem, putting the imageurl in the template xml should work, isn't it normal?

I just tested with local image path and i don't know about http, or https url

if you take a look at these codes, I use some pattern like #1# to replace what I want, when I wrote this package I checked many xml like that and I don't see any problem, I suggest you use some XML editor like the android studio to create XML, then add your pattern to XML and when you want to post notification replace the pattern with anything you want like(big text,text,image,type and etc...)

Then, if there is no problem, putting the imageurl in the template xml should work, isn't it normal?

okaay....got it. Thank you for the quick reply. Have a nice day!


Get.changeTheme not updating on change route

Describe the bug
When calling Get.changeTheme(theme) followed by Get.off(route), the new pushed route has the old theme until a rebuild is made.
Reproduction code
import 'package:flutter/material.dart';
import 'package:get/get.dart';

void main() {
  runApp(MyApp());
}

ThemeData theme1 = ThemeData(
  backgroundColor: Colors.red,
);
ThemeData theme2 = ThemeData(
  backgroundColor: Colors.green,
);

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return GetMaterialApp(
      theme: theme1,
      home: Page1(),
    );
  }
}

class Page1 extends StatelessWidget {
  const Page1({Key key}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return Center(
      child: Container(
        height: 200,
        width: 200,
        child: RaisedButton(
          child: Text('navigate'),
          onPressed: () {
            Get.changeTheme(theme2);
            Get.off(Page2());
          },
        ),
      ),
    );
  }
}
class Page2 extends StatelessWidget {
  const Page2({Key key}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return Center(
      child: Container(
        height: 200,
        width: 200,
        color: Get.theme.backgroundColor,
      ),
    );
  }
}

To Reproduce
Steps to reproduce the behavior:

Click the navigate button

Expected behavior
The new route should come with updated theme (color be green) but the color only updates after rebuilding widget.
Flutter Version:
1.23.0
Getx Version:
3.15.0
Describe on which device you found the bug:
Android emulator
Minimal reproduce code
Provide a minimum reproduction code for the problem

According to the doc, Get.theme is immutable.
If you want to work with dynamic themes, you should use
color: context.theme.backgroundColor,

I couldn't find this in the docs, but it worked with context.theme
Thank you :)


🐛 [Github] Link in the "About" section on Github does not work anymore

The link https://firebase.google.com/docs/flutter/setup does not work anymore.

@jaredsburrows
I tried the link and it is working as expected.
https://github.com/firebase/flutterfire/assets/67046386/817a8540-da98-46cb-9362-1ddccdf53741
The link opens the screen / page properly:
https://firebase.google.com/docs/flutter/setup?platform=android
Will close as WAI. If you disagree, write in comments and I'll reopen it.

Thanks. Must have been bad cache.


CodeBuffer should have the same signature as StringBuffer

CodeBuffer should have the same signature as StringBuffer.
It should not return itself after writing... to it.
Assigning to me: I will probably do it when changing from "add" to "write".

I don't think this is relevant after the emitter refactorings.


Does the migration completed?

Hello @AlexKrupko , it's really excited to find a batarang chrome extension here, it could be really helpful. But I wonder whether it is completed or not. I see some deprecated API such as chrome.pageAction.show in the background.js.

Hi @Lefschetz , yes, it still continues using some of the deprecated features. I didn't rewrite the extension, I took the code from the original extension and just migrated it to manifest v3 with fixing some critical issues.
So far it's working, I see no reason to spend time for rewriting the code, even if there are some deprecated features.


Add Thread-safe Block Index Read

Current dependencies on/for this PR:

feature/proof-of-stake

PR #846   👈

This stack of pull requests is managed by Graphite.


Cassandra CqlSession effect wrapper

The PR https://github.com/avast/scala-server-toolkit/pull/74 added functionality to create an instance of com.datastax.oss.driver.api.core.CqlSession
We should probably not return this object directly, but a wrapper which returns F[_]: Async (or whatever), so that it's interoperable with cats-effect.
We would need to interop with CompletionStage which is what CqlSession returns, thus this is blocked on https://github.com/avast/scala-server-toolkit/issues/87

I think this calls for a standalone library which we can use in SST. I don't want to provide such wrappers in SST because the surface of SST would become too large - it is not supposed to become a framework. The main goal of SST is to provide initialization and integration of other libraries. So if there is a library that provides wrapper around CqlSession we can create another module and make it work nicely with the whole ecosystem.

I see :thinking:


[HOLD for payment 2023-07-21] [$1000] Fix upstream ESLint bug

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Problem
We are affected by this upstream bug in ESLint: https://github.com/eslint/eslint/issues/15261
Solution
Fix it!
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01e054af762caa3272
Upwork Job ID:<PHONE_NUMBER>541461504
Last Price Increase: 2023-07-08

Not sure why we got both @tjferriss and @alexpensify, so I'm going to unassign @alexpensify since he was assigned second

@roryabraham It seems like there is a fix already in progress in upstream repository.

According to this comment, that fix will probably come with v9.0.0

Contributor details
Your Expensify account email<EMAIL_ADDRESS>Upwork Profile Link: https://www.upwork.com/freelancers/~0134da57005be11004

Proposal
Please re-state the problem that we are trying to solve in this issue.
An ESLint bug about defining more that one 'importNames' for the same 'name' with 'no-restricted-imports' rule. Only last 'importNames' is checked for the same 'name'.
What is the root cause of that problem?
Here only StatusBar will be checked becuase of the ESLint bug,
https://github.com/Expensify/App/blob/60f48175c9e3342ef192d878218820e5ed5a76cd/.eslintrc.js#L19C1-L42C7
What changes do you think we should make in order to solve the problem?
All the separate 'importNames' can be converted into a single 'importNames'. And message can be modified accordingly.
  'no-restricted-imports': [
      'error',
      {
          paths: [
              {
                name: 'react-native',
                importNames: ['useDimensions', 'StatusBar', 'TouchableOpacity', 'TouchableWithoutFeedback', 'TouchableNativeFeedback', 'TouchableHighlight'],
                message: `If you're using following components from react native:\n
                             Touchable* => please use PressableWithFeedback and/or PressableWithoutFeedback from src/components/Pressable instead\n
                             Statusbar => please use useWindowDimensions from src/hooks/useWindowDimensions instead\n
                             useDimensiongs => please use useWindowDimensions from src/hooks/useWindowDimensions instead
                         `,
              }
          ],
      },
  ],

What alternative solutions did you explore? (Optional)
None

Proposal
Please re-state the problem that we are trying to solve in this issue.
Fix upstream ESLint bug
What is the root cause of that problem?
I've gone through the code of the rule from eslint repo and I found it that rule will hold only importName of the late value meaning the previous imports are overridden.
What changes do you think we should make in order to solve the problem?
Maybe add a new rule inside eslint-config-expensify which solves the problem wanted. Restricting imports from specific modules (for now).
I have written a rule to fix this, which seems working fine.
Implementation Demo:
https://github.com/Expensify/App/assets/59088937/ebb2d0fd-8dd2-458a-9b94-129d04525dba

What alternative solutions did you explore? (Optional)
NA

@allroundexperts imo https://github.com/eslint/eslint/pull/16196 is kind of yucky since it introduces an "allowlist" approach where only certain imports are explicitly allowed, instead of a "blocklist" approach where only certain imports are explicitly disallowed. What do you think?

Proposal
Please re-state the problem that we are trying to solve in this issue.
Not all restricted react-native imports are restricted due to ESLint bug.
What is the root cause of that problem?
An issue with ESLint concerning the no-restricted-imports rule. When more than one importNames are defined for the same name, only the last importNames is verified, which is the root cause of this bug.
What changes do you think we should make in order to solve the problem?
All distinct importNames can be combined into a single importNames, with an appropriate modification in the accompanying message. The message is formatted in a list and joined with \n for better visibility.
Since ESlint team is working on the fix in the upstream repository I’m not sure if it’s worth the effort to fix the rule on our side.
'no-restricted-imports': [
    'error',
    {
        paths: [
            {
                name: 'react-native',
                importNames: ['useWindowDimensions', 'TouchableOpacity', 'TouchableWithoutFeedback', 'TouchableNativeFeedback', 'TouchableHighlight', 'StatusBar'],
                message: [
                    "For 'useWindowDimensions', please use 'src/hooks/useWindowDimensions' instead.",
                    "For 'TouchableOpacity', 'TouchableWithoutFeedback', 'TouchableNativeFeedback', 'TouchableHighlight', please use 'PressableWithFeedback' and/or 'PressableWithoutFeedback' from 'src/components/Pressable' instead.",
                    "For 'StatusBar', please use 'src/libs/StatusBar' instead.",
                ].join('\n'),
            },
        ],
    },
],

What alternative solutions did you explore? (Optional)
None

Ok, since @HLEDYA proposed prettymuch the same fix as @blazejkustra and did so first, I'm going to assign this job to @HLEDYA to implement the workaround in our eslint config

But having a single message for all the imports seems good?!

Thanks @roryabraham, PR will be submitted soon.
@b4s36t4, imo it is a workaround until the fix comes from ESLint team. Btw, for imports from different paths, distinct messages can still be created. But for the imports from the same path as in our case, this workaround needs to be applied. Since we don't have tons of import rules for the same path, it seems reasonable.

The PR is submitted and under review.

Upworks offers sent to @allroundexperts and @HLEDYA. This job is eligible for a 50% #urgency bonus.

Payment to @HLEDYA tomorrow assuming no regression.

Hi @tjferriss, afaik there is none.

@tjferriss this is ready to pay

Paid!

I received the payment, thank you all.

@tjferriss Can you please summarize the appropriate individual payments for all parties involved in this issue? This is holding up @rushatgabhane's NewDot payments. More information on this compliance process in Slack.

Updated payment comment https://github.com/Expensify/App/issues/22459#issuecomment-1635954727

Reviewed details for @allroundexperts . These details are accurate based on summary from Business Reviewer and are now approved for payment in NewDot.


Fix messages for file lookup issues - error scenario

Scenario where multiple files (all) are not found only show a single message.
I feel its useful for debugging to keep the message if only 1 file is not found. Happy to remove if others don't agree.
No changes where at least 1 is found.
See PR https://github.com/mochajs/mocha/pull/3650 for full details
Scenarios
Nothing given
Command: mocha
New output:
Error: Cannot find any files matching pattern "test/"

Previous output:
Error: Cannot find any files matching pattern "test/"

Nothing found
Command: mocha -R json-stream --no-config "test/integration/fixtures/glob/**/*-none.js"
New output:
Error: Cannot find any files matching pattern "test/integration/fixtures/glob/**/*-none.js"

Previous output:
Error: Cannot find any files matching pattern "test/integration/fixtures/glob/**/*-none.js"

Multiple not found
Command: mocha -R json-stream --no-config "test/integration/fixtures/glob/**/*-none.js" "test/integration/fixtures/glob/**/*-none-again.js"
New output
Error: No test files found

Previous output
Error: Cannot find any files matching pattern "test/integration/fixtures/glob/**/*-none.js"
Error: Cannot find any files matching pattern "test/integration/fixtures/glob/**/*-none-again.js"

Applicable issues
(semver-patch)

Coverage increased (+0.003%) to 90.773% when pulling 5d49ff31cf4b34a1c72cd82a6a3009271c9274b6 on craigtaub/fixNoisyMessages2 into cb1c6b013c6c789e190081801bc628a18f81ba1b on master.


Turkish Character issue

Hello there I have Turkish character problem. This characters look very bad on this theme "ÇçĞğİıŞş". Is there any solution for that?

***SOLVED

Edit assets/css/atlantis.css file. Change this "font-family: 'Lato', sans-serif;" to "font-family: 'Open Sans', sans-serif;" .
Edit assets/css/atlantis.min.css file. Find first "Lato" word and change to "Open Sans" and save file.
Clear browser history and refresh page.


[CR0025] Replace default dropdown with custom dropdown

Problem defination:
Pecha.org is currenly using default html dropdown which does not support custom css stying like Padding and Margin etc,
Which is  required when we display tibetan text
Implementation steps

[x] list all default dropdown
[x] replace it with custom dropdown

could only check in the "share menu" of existing sheets. Don't know if there are at any other locations on the website


Create bknrd.txt

Please add the domain bknrd.schule to this list. Thanks!

@Zugvergleich Please provide us with a proof, which shows that the school recognizes the domain you are submitting as an official email domain for the students.
You can provide us with any of the following:

A copy of instruction (a hard-copy or electronic one) given to you by the school that tells you to use an email address in this email domain
An URL of a page at the official school website that says that students use this email domain
A reference to a recognized journal where an article was published by someone from the school, with explicit affiliation to the school, and e-mail address from the email domain
A WHOIS data that link the email domain to the school somehow
A copy of official letter from the school representative saying that the school provides e-mail addresses in this domain to the students
Anything else that can prove this fact in similar way, i.e. can be verified independently.

https://berufskolleg-nord.de/wp-content/uploads/2024/08/Nutzungsvereinbarung-und-datenschutzrechtliche-Einwilligung08_23.pdf
In this document you can find on page 1 the domain @bknrd.schule that I requested.

@Zugvergleich  Thank you for the clarifications. Pull request merged. Please start requesting the licenses in about an hour, to let the changes to propagate through our system.


Sonobuoy timeout needs to be more clear

What steps did you take and what happened:
A test run which takes a long time and times out; e.g. sonobuoy run --e2e-focus=sig-storage will timeout after 3h but it is hard to diagnose the problem.
For instance, the # of tests to run, that have run, etc by ginkgo don't add up as expected. In addition, the timeout lines are buried in the middle of Sonobuoy/e2e logs because there are numerous cleanup/followup actions that take place.
What did you expect to happen:
If there is a Sonobuoy timeout we need to make sure that it is:

highly visible; preferably the last thing logged as it exits
the status reflects the timeout if you run sonobuoy status
the log lines for the timeout are errors (looks like they are info lines now)

Anything else you would like to add:
xref #880

After investigating this there are also some inconsistencies with how we handle these timeouts that I want to remedy. As a bonus, it simplifies some of the logic.
To fixup the timeout behavior I want to roll back part of https://github.com/heptio/sonobuoy/commit/4e4513dbdc84e742b1274ab0fde9f21eb317d77b which added the graceful shutdown of plugins.
The issue it was trying to solve was related to ensuring that when things get stalled/wedged we want to be able to get diagnostic information.
The problem now is that if you look at the timeout flow, the graceful shutdown causes the pods to get cleaned up (i.e. deleted) before the typical cluster queries take place. This means that we lose out on the json data of the pods as well as the podlogs themselves.
In addition, it is unintuitive to any caller of Run that the pods in questino will get deleted if and only if there is a timeout.
The graceful shutdown also is handled by the conformance test image in such a way that it will stop running the tests and report back what it already has done. This has some consequences:

you do end up getting the e2e.log file as part of the results from the plugin (a good thing)
you also get the junit.xml file with actual junit results in it which seems like a good thing until you realize the the junit doesn't properly report this timeout behavior. So you get in a situation where the logs say (for instance) 276 tests will run but at the end of the logs only 100 ran with no failures. Then the junit just says 100 tests ran and passed. This false positive is very confusing to users and the fact that it was a timeout is buried in the logs and will surely be missed by automation.

IMO the appropriate solution here is:

do away with the graceful shutdown. If you miss the deadline, you miss the deadline.
since we dont have graceful shutdown

it is more clear which plugins timed out and which reported results in time. (Graceful shutdown complicated this because the aggregator would try and kill the pods, leading to results being forced from the plugin and then you dont know if the results are a consequence of the plugin finishing normally or the timeout itself. You just can't trust the results)
the pods that timed out will stick around and we can get the logs/json as normal
now that we know very clearly which pods timed out, we can mark their status as such on the status annotations

FWIW I just want to roll back the aggregators handling of this; I don't care to roll back the signal handling of the Sonobuoy worker or job deletion defaults. I'm fine with there being a value for that.


Handle exception when jar does not exist while updating CDK property in admin client

Description
If a jar does not exist in the jars bucket, the CDK will fail trying to get the latest version of the jar. When using the admin client, this breaks out of the client and displays an exception.
Steps to reproduce

Deploy a test instance
Rename one of the jars in the jars bucket
Run the admin client and change a property that requires a CDK deployment
See exception

Expected behaviour
We should handle the exception and display a message along the lines of CDK deployment failed: One or more jars could not be found in the jars bucket

It looks like there are a few parts to this.

The admin client doesn't upload any updated jars to the bucket when it's applying properties changes in the CDK. That's probably not really a problem, as if the jars have changed you want to upgrade the instance explicitly.

Any time the CDK is invoked, it will check whether the version matches the deployed version number. If it doesn't match, the CDK deployment will fail at the beginning. It looks like the way we found this problem originally can only occur in a development environment, where we redeploy with the same version number.

It sounds like when the CDK fails when applying properties changes, the admin client fails completely and terminates? That does sound like a problem, although we may resolve it when we rework the admin client.

Should we take this out of the milestone and deal with it when we come to rework the admin client? We probably want some general solution so that when any operation fails it's handled in a consistent way.

I can't reproduce this, and from the code I can't see how the client would terminate on a failure in the CDK. I'm going to close this, but we could revisit it if we get a log that shows a problem in the admin client.


[Snyk] Security upgrade node from 14.18.3-alpine to 14.19.0-alpine

This PR was automatically created by Snyk using the credentials of a real user.Keeping your Docker base image up-to-date means you’ll benefit from security fixes in the latest version of your chosen image.
Changes included in this PR

forms-flow-web/Dockerfile

We recommend upgrading to node:14.19.0-alpine, as this image has only 4 known vulnerabilities. To do this, merge this pull request, then verify your application still works as expected.
Some of the most important vulnerabilities in your base image include:

Severity
Priority Score / 1000
Issue
Exploit Maturity

500
CVE-2022-28391 SNYK-ALPINE315-BUSYBOX-2440607
No Known Exploit

500
CVE-2022-28391 SNYK-ALPINE315-BUSYBOX-2440607
No Known Exploit

400
Loop with Unreachable Exit Condition ('Infinite Loop') SNYK-ALPINE315-OPENSSL-2426331
No Known Exploit

400
Loop with Unreachable Exit Condition ('Infinite Loop') SNYK-ALPINE315-OPENSSL-2426331
No Known Exploit

400
Out-of-bounds Write SNYK-ALPINE315-ZLIB-2434420
No Known Exploit

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
🛠 Adjust project settings

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Learn about vulnerability in an interactive lesson of Snyk Learn.

node was updated to 14.20.0-alpine


Identifier Triplets RFC-001

Description
Proposing RFC-001, identifier triplets
Additional Notes

added a new rfc template for future proposals

<--------------------- REMOVE THE LINES BELOW BEFORE MERGING --------------------->
Checklist
Please review and complete the following checklist before submitting your pull request:

[x] I have ensured that the pull request is of a manageable size, allowing it to be reviewed within a single session.
[x] I have reviewed my changes to ensure they are clear, concise, and well-documented.
[x] I have updated the documentation, if applicable.
[x] I have added or updated test cases to cover my changes, if applicable.
[x] I have minimized the number of reviewers to include only those essential for the review.

Checklist for Reviewers
Please review and complete the following checklist during the review process:

[ ] The code follows best practices and conventions.
[ ] The changes implement the desired functionality or fix the reported issue.
[ ] The tests cover the new changes and pass successfully.
[ ] Any potential edge cases or error scenarios have been considered.

During our review in the Record Linker <> DIBBs sync on 11/20, here are the questions/feedback Marcelle and Brandon brought up:
* Because the `"authority"` and `"type"` fields are optional, they recommended investigating how missingness in these fields impacts the performance of this approach. For example, if `"authority"` and `"type"` fields are missing a high percentage of the time, how does this concatenation approach perform in terms of match accuracy? Does this approach handle cases of data missingness with sufficient performance?

The type and value fields will be required, only authority is optional.  I'll double-check the RFC to make sure that is clear.

* Does this approach apply to MRN as well? Or do we have a separate feature matching function for MRN, given it is an important feature with high predictive power?

The MRN, SSN and DRIVERS_LICENSE features would all be removed and replaced with IDENTIFIER:MR, IDENTIFIER:SS and IDENTIFIER:DL respectively.

* If `IDENTIFIER` and `IDENTIFIER:TYPE` will become newly supported features for matching, what will calculating their log odds look like? E.g., will there be one general log odds score for `IDENTIFIER` or one for each `IDENTIFIER:TYPE`?

Yeah this is an interesting point, I think a log odds score for IDENTIFIER might be difficult or impossible to create, so maybe that shouldn't be supported in the log odds type functions.  Only IDENTIFIER:X should work.

* For `"authority"` value, Marcelle recommended looking into `Identifier[0].system` in addition to what you document in the RFC, which looks like `Identifier[0].assigner.display`. This should help circumvent the lack of standardization in cases like `"California"` vs. `"CA"` vs. `"DMV of California"`.

👍

* Be sure to use FHIR r4, not r5 (no EHRs currently use r5; one of the links in the RFC redirects to r5 docs)

👍


Adding equals and hashcode to ArchaiusModule

NetflixOSS » archaius » archaius-pull-requests #189 FAILURE
Looks like there's a problem with this pull request

archaius-pull-requests #355 FAILURE
Looks like there's a problem with this pull request

archaius-pull-requests #356 SUCCESS
This pull request looks good

ArchaiusModule should not be dedupped as it is only used for customizing archaius.  You should be able to install multiple of these, each with different customizations.  This follows the approach used by ServletModule

NetflixOSS » archaius » archaius-pull-requests #190 SUCCESS
This pull request looks good

archaius-pull-requests #357 SUCCESS
This pull request looks good

NetflixOSS » archaius » archaius-pull-requests #191 SUCCESS
This pull request looks good


Start FetcherDB from Scratch on Startup

store in .cache/ or whatever

https://github.com/CounterpartyXCP/counterparty-core/pull/2663/files/f00b2b50df49c1000523d7025eb400d44f6c82a4..9025e8cf3f3bcf19b8dc715e339a159ce0c64718


Added unassign_users.py v1

Why:

Closes #5709
What's being changed: Added unassign_users.py as per required.

Need to add authorized tokens to authenticate using an access key at line 10 and you are good to go

Hey @patel-aum
Thanks for contributing this script. But there is some issue with this script.

The script is not complete
The Github API do not provides assigned_at value

[
{
    "url": "https://api.github.com/repos/codinasion/program/issues/5657",
    "repository_url": "https://api.github.com/repos/codinasion/program",
    "labels_url": "https://api.github.com/repos/codinasion/program/issues/5657/labels{/name}",
    "comments_url": "https://api.github.com/repos/codinasion/program/issues/5657/comments",
    "events_url": "https://api.github.com/repos/codinasion/program/issues/5657/events",
    "html_url": "https://github.com/codinasion/program/issues/5657",
    "id":<PHONE_NUMBER>,
    "node_id": "I_kwDOIDzQVs5YtBjL",
    "number": 5657,
    "title": "Write a Swift program to split string into array of words",
    "user": {
      "login": "harshraj8843",
      "id": 54644599,
      "node_id": "MDQ6VXNlcjU0NjQ0NTk5",
      "avatar_url": "https://avatars.githubusercontent.com/u/54644599?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/harshraj8843",
      "html_url": "https://github.com/harshraj8843",
      "followers_url": "https://api.github.com/users/harshraj8843/followers",
      "following_url": "https://api.github.com/users/harshraj8843/following{/other_user}",
      "gists_url": "https://api.github.com/users/harshraj8843/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/harshraj8843/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/harshraj8843/subscriptions",
      "organizations_url": "https://api.github.com/users/harshraj8843/orgs",
      "repos_url": "https://api.github.com/users/harshraj8843/repos",
      "events_url": "https://api.github.com/users/harshraj8843/events{/privacy}",
      "received_events_url": "https://api.github.com/users/harshraj8843/received_events",
      "type": "User",
      "site_admin": false
    },
    "labels": [
      {
        "id":<PHONE_NUMBER>,
        "node_id": "LA_kwDOIDzQVs8AAAABEH5vFQ",
        "url": "https://api.github.com/repos/codinasion/program/labels/good%20first%20issue",
        "name": "good first issue",
        "color": "FBCA04",
        "default": true,
        "description": "Good for newcomers"
      },
      {
        "id":<PHONE_NUMBER>,
        "node_id": "LA_kwDOIDzQVs8AAAABEH-TVQ",
        "url": "https://api.github.com/repos/codinasion/program/labels/Swift",
        "name": "Swift",
        "color": "006B75",
        "default": false,
        "description": "swift related"
      }
    ],
    "state": "open",
    "locked": false,
    "assignee": {
      "login": "sahilg2931",
      "id": 54029096,
      "node_id": "MDQ6VXNlcjU0MDI5MDk2",
      "avatar_url": "https://avatars.githubusercontent.com/u/54029096?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/sahilg2931",
      "html_url": "https://github.com/sahilg2931",
      "followers_url": "https://api.github.com/users/sahilg2931/followers",
      "following_url": "https://api.github.com/users/sahilg2931/following{/other_user}",
      "gists_url": "https://api.github.com/users/sahilg2931/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/sahilg2931/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/sahilg2931/subscriptions",
      "organizations_url": "https://api.github.com/users/sahilg2931/orgs",
      "repos_url": "https://api.github.com/users/sahilg2931/repos",
      "events_url": "https://api.github.com/users/sahilg2931/events{/privacy}",
      "received_events_url": "https://api.github.com/users/sahilg2931/received_events",
      "type": "User",
      "site_admin": false
    },
    "assignees": [
      {
        "login": "sahilg2931",
        "id": 54029096,
        "node_id": "MDQ6VXNlcjU0MDI5MDk2",
        "avatar_url": "https://avatars.githubusercontent.com/u/54029096?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/sahilg2931",
        "html_url": "https://github.com/sahilg2931",
        "followers_url": "https://api.github.com/users/sahilg2931/followers",
        "following_url": "https://api.github.com/users/sahilg2931/following{/other_user}",
        "gists_url": "https://api.github.com/users/sahilg2931/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/sahilg2931/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/sahilg2931/subscriptions",
        "organizations_url": "https://api.github.com/users/sahilg2931/orgs",
        "repos_url": "https://api.github.com/users/sahilg2931/repos",
        "events_url": "https://api.github.com/users/sahilg2931/events{/privacy}",
        "received_events_url": "https://api.github.com/users/sahilg2931/received_events",
        "type": "User",
        "site_admin": false
      }
    ],
    "milestone": null,
    "comments": 1,
    "created_at": "2022-12-10T08:13:04Z",
    "updated_at": "2022-12-31T14:57:00Z",
    "closed_at": null,
    "author_association": "MEMBER",
    "active_lock_reason": null,
    "body": "### Description\n\nWrite a Swift program to split string into array of words\n\n```\nInput  : \"hello world\"\nOutput : [\"hello\", \"world\"]\n```\n\nHow to contribute\n\n- Save the solution in `SplitStringIntoArrayOfWords.swift` file\n- Add `SplitStringIntoArrayOfWords.swift` file in `program/split-string-into-array-of-words` folder\n\n\n    ",
    "reactions": {
      "url": "https://api.github.com/repos/codinasion/program/issues/5657/reactions",
      "total_count": 0,
      "+1": 0,
      "-1": 0,
      "laugh": 0,
      "hooray": 0,
      "confused": 0,
      "heart": 0,
      "rocket": 0,
      "eyes": 0
    },
    "timeline_url": "https://api.github.com/repos/codinasion/program/issues/5657/timeline",
    "performed_via_github_app": null,
    "state_reason": null
  },
...
]

The way I am thinking of the solution is as follows:-

GET all good-first-issues from the Github API
GET https://api.github.com/repos/codinasion/program/issues?labels=good first issue

Check for the length of assignees. If length > 0

Get issue events from the Github API
GET https://api.github.com/repos/codinasion/program/issues/5657/events

Find the latest assignment event
"event": "assigned",

Check event time and compare from time. now
"created_at": "2022-12-31T14:57:00Z",

If the assignment is older than a specific time period unassign the user
https://docs.github.com/en/rest/issues/assignees?apiVersion=2022-11-28#remove-assignees-from-an-issue


Bug: Chrome plays two videos at the same time

With the switch to single page for a lesson Chrome has the ability to play two videos at the same time.
Probably need to ensure that the old video is really stopped before switching.

Fixed in latest commit


Migrate from no-implicit-casts to strict-casts

Closed by https://github.com/google/sync_http.dart/pull/34 – sorry I missed this, @srawlins !


docker: configurable git user / group ID build args

Per #23632, there are several issues with the hardcoded values (1000:1000) in the existing Docker installation flow.
This change adds GIT_GID and GIT_UID env vars to the Dockerfile. This allows the builder to optionally set custom user and group IDs for the git user. If the values are not set, it falls back to the original behavior.

Is this still a draft PR?

Closing to declutter open PRs


[Question] lambdarank and the label_gain parameters

Hello
I have a dataset with 3 classes, but classes 1 and 2 represent only 1/3=2/6 of the data and class 3 represents 2/3=4/6. I'm using objective=mutliclass.
Shall add a weight for class 1 and 2 labels so it is balance with class 3?
Is it done via the 'lambdarank' and the 'label_gain' parameters? like label_gain=4,4,1
Thanks
--w
ref - lambdarank and the label_gain parameters

lambdarank is not a parameter... it's a loss function (a value to be passed to the objective parameter) for learning-to-rank applications.
label_gain is not relevant for multiclass classification... it's only used for learning-to-rank with lambdarank, as described in the docs at https://lightgbm.readthedocs.io/en/latest/Parameters.html#label_gain.
Some strategies for dealing with class imbalance for multiclass classification

re-weight the data (as you suggested), by providing training weights
use multiclassova objective + is_unbalance=true (docs link)
if you have a lot of features, try using very low values of feature_fraction (like <= 0.1) ... this might help in the case where some subset of features are much more helpful for one of the minority classes
use LGBMClassifier from the Python package and its class_weight parameter (docs)

You can also try searching around this project's issue board, Stack Overflow, Kaggle boards, etc. ... there's much discussion about how to deal with class imbalance for multiclass problems using gradient boosted trees.
Linking some related conversations:

https://github.com/microsoft/LightGBM/issues/3702
https://github.com/microsoft/LightGBM/issues/5399
https://github.com/dmlc/xgboost/issues/7688

Super - thanks James
I will try those over the weekend     On Thursday, August 15, 2024 at 10:02:32 PM PDT, James Lamb @.***> wrote:
lambdarank is not a parameter... it's a loss function (a value to be passed to the objective parameter) for learning-to-rank applications.
label_gain is not relevant for multiclass classification... it's only used for learning-to-rank with lambdarank, as described in the docs at https://lightgbm.readthedocs.io/en/latest/Parameters.html#label_gain.
Some strategies for dealing with class imbalance for multiclass classification

re-weight the data (as you suggested), by providing training weights
use multiclassova objective + is_unbalance=true (docs link)
if you have a lot of features, try using very low values of feature_fraction (like <= 0.1) ... this might help in the case where some subset of features are much more helpful for one of the minority classes
use LGBMClassifier from the Python package and its class_weight parameter (docs)

You can also try searching around this project's issue board, Stack Overflow, Kaggle boards, etc. ... there's much discussion about how to deal with class imbalance for multiclass problems using gradient boosted trees.
Linking some related conversations:

#3702
#5399
dmlc/xgboost#7688

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>


IPython subtasks run always only with 1 core

Hi there, this is the script I use to submit bcbio to our cluster with slurm as job scheduler:

#!/usr/bin/env bash
#SBATCH --job-name=bcbio
#SBATCH --qos=medium
#SBATCH--output=multithread-%j.out
export PATH=/bcbio_ii2/local/bin:$PATH
bcbio_nextgen.py /scratch/slurm/cancer-dream-syn3/config/cancer-dream-syn3.yaml -n 28 -t ipython -s slurm -q work -r qos=medium

And everything runs smoothly except for the fact that all subtasks like bwa etc. are only called with one available core:

[2017-11-03T19:48Z] cn-12: System YAML configuration: /bcbio_ii2/local/share/bcbio/galaxy/bcbio_system.yaml
[2017-11-03T19:48Z] cn-12: Resource requests: bwa, sambamba, samtools; memory: 32.00, 32.00; cores: 16, 16, 16
[2017-11-03T19:48Z] cn-12: Configuring 1 jobs to run, using 1 cores each with 32.00g of memory reserved for each job
2017-11-03 20:48:44.640 [IPClusterStart] Using existing profile dir: u'/scratch/slurm/log/ipython'

I especially do not understand whats going on between those two lines:

[2017-11-03T19:48Z] cn-12: Resource requests: bwa, sambamba, samtools; memory: 32.00, 32.00; cores: 16, 16, 16
[2017-11-03T19:48Z] cn-12: Configuring 1 jobs to run, using 1 cores each with 32.00g of memory reserved for each job

The first line represents what I defined in the global resources section and then bcbio seems to fall back into a 1-core mode?
Any idea whats going wrong here?
Thank you!

Hi @leiendeckerlu,
Further up from the bottom it should tell you what step it is trying to currently do. The first step in the process is to submit a job to the scheduler, so we can figure out what kind of compute the compute nodes have. That might be what you are seeing.
If it is doing a processing step like alignment, then you might have a file in the checkpoint directory that is stopping it from submitting the job.

Thanks for the example and help debugging all. It looks like your resource specification in bcbio_system.yaml asks for 32Gb of memory per core (along with 16 cores). It seems like then bcbio determines your machine has 32Gb of total memory and scales the core request down to 1 to avoid over-subscribing memory. You want to set the memory in bcbio_system.yaml to be memory per core, so probably 2Gb in this case to maximize cores (assuming I'm understanding your system resources right). Hope this fixes it for you.

Thanks for your feedback!
@chapmanb  Indeed I'm using cluster nodes that (should) have 32 GB per core available, so that should not be a limitation. My idea behind setting the memory to 32 GB per core was to determine the overall real memory usage for particular steps in the pipeline and thus adjust the core/memory bcbio_systems.yaml for the respective tool. I observe the same phenomena also with 8 GB per core. Why does bcbio think I only have 32 GB in total if I tell him that indeed there are 512 GB available?
@roryk The first step that is not using multiple threads is bwa mem (since I guess the fastq indexing
(grabix) is anyway not multithreaded). Even when running the above mentioned commands in a completely fresh directory it does not scale properly.

Thanks for the additional details. It sounds like bcbio might not be detecting the available memory on your cluster machines. If you look at provenance/system-ipython-queue.yaml that should tell you what bcbio thinks each machine has based on submitting a test job. If you have a heterogeneous cluster this might not identify the memory/core allocation you want. You can specify this manually if not:
https://bcbio-nextgen.readthedocs.io/en/latest/contents/parallel.html#determining-available-cores-and-memory-per-machine
Hope this helps explain what is going on and setting it provides a fix.

Mhm, so this is what my provenance/system-ipython-queue.yaml looks like:

cores: 1
memory: 0.0009765625
name: slurm_machine

and this is what /local/share/bcbio/galaxy/bcbio_system.yaml looks like:

default:
memory: 4g
cores: 16
jvm_opts: ["-Xms750m", "-Xmx3500m"]
bwa:
cores: 16

So it seems like it is not using the settings in bcbio_system.yaml ?

Thanks for posting all of the details. The provenance/system-ipython-queue.yaml file is what bcbio thinks each machine has. Either your SLURM setup is misconfigured in some way or something else is going wrong in looking at the resources.
The bcbio_system.yaml setup you have there doesn't tell bcbio what resources a machine has, it tells it what you want to use. To tell it the correct resources, please add a resources section to your configuration as I mentioned in the last comment:
https://bcbio-nextgen.readthedocs.io/en/latest/contents/parallel.html#determining-available-cores-and-memory-per-machine
Hopefully that'll fix it for you.

@chapmanb Ah well, I misunderstood your previous statement, sorry for that!
Adding a resources section in my config indeed did the job, everything is running nicely now.
Thank you!


Other flags is disabled in SDK-style project

In project properties / build settings, the "Other flags" input field is disabled in a new SDK-style project. Note, this project does not target .NET Core, but net45.
Expected behavior
I expect to be able to use it. I use the flag --standalone to have a single assembly to install on Sql Server (SqlCLR).
Actual behavior
Greyed out in UI, and I've found no way to pass the flags in the .fsproj either
Known workarounds
None. I'd be happy to just put my flags directly in the fsproj, but I don't know how.
Related information

Win 10, VS 15.9.4

I think this is fallout from #3924. Because both use the same project system UI, there isn't a good way to say, "this is applicable for this target, but if you switch your target somehow, it's no longer applicable". Unfortunately, this problem is not unique to F#. The property pages are very old and there is zero target-based UI lightup based on a dimension (i.e., TFM) of a project.
Not sure if this fits your specific needs, but the following project file does build:
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net45</TargetFramework>
    <OtherFlags>--standalone</OtherFlags>
  </PropertyGroup>

  <ItemGroup>
    <Compile Include="Program.fs" />
  </ItemGroup>

</Project>

Thanks!
Everything does seem to work, but I get a few warnings from compilation that I didn't get before converting to the new proj format.

    FSC : * warning : PDB file could not be read and will be ignored: Object reference not set to an instance of an object. [E:\dev\vuf\VufSql\VufSql.fsproj]
    FSC : warning FS2011: Assembly 'System.ServiceModel.Internals' was referenced transitively and the assembly could not be resolved automatically. Static linking will assume this DLL has no dependencies on the F# library or other statically linked DLLs. Consider adding an explicit reference to this DLL. [E:\dev\vuf\VufSql\VufSql.fsproj]
    FSC : warning FS2011: Assembly 'SMDiagnostics' was referenced transitively and the assembly could not be resolved automatically. Static linking will assume this DLL has no dependencies on the F# library or other statically linked DLLs. Consider adding an explicit reference to this DLL. [E:\dev\vuf\VufSql\VufSql.fsproj]
    FSC : * warning : PDB file could not be read and will be ignored: Object reference not set to an instance of an object. [E:\dev\vuf\VufSql\VufSql.fsproj]

@rojepp
The OtherFlags setting is still wired up in the F# targets:  https://github.com/Microsoft/visualfsharp/blob/master/src/fsharp/FSharp.Build/Microsoft.FSharp.Targets#L289

@KevinRansom Thanks. I'm using it and it does seems to work.
The issue is still there though, in that this doesn't work in UI.

Ahhh!!!! the entry field is greyed out in SDK projects.  It seems like that should be fixed then :-)

@KevinRansom This shouldn't be labeled as the legacy project system. It's for .NET SDK-style projects. Do we control that UI regardless? If so we should have a different label.

Since we own the property pages but the wireup needs to happen in the .NET project system, we'll keep the current label and re-add External


Set a correct description for io.spine.protodata.protodata-fat-cli artifact

We assemble protodata-fat-cli artifact from many source JARs. Therefore, in the resulting fat JAR, one of the metadata fields is apparently copied from some Google library, causing the description of our artifact to read
PROTO library for proto-google-common-protos

which it is obviously not.
In scope of this issue we should put a proper description instead of the current one.

Removed the wrong description via manual editing in GitHub.


Will the training code be released

hi, thank you for the good work, will the training code be released also? thanks!

Sorry, but we don't plan to release the train code.


METRON-116 Automated Testing and Rename of Fastcapa

Changes

Created automated integration test for fastcapa
Renamed packet capture to fastcapa to distinguish it from pycapa.
Improved fastcapa README to show use of test environment.
Made sensor-test-mode role configurable so that it could be used in deployment/vagrant/fastcapa-vagrant.

Validation
Run the following commands.  If the environment is deployed successfully, then the automated integration test has passed successfully.
cd metron-deployment/vagrant/fastcapa-vagrant
vagrant up

Attempting to trigger a Travis CI build by closing and reopening the PR.

+1, this looks good.

+1


Lock to RELEASE version of com.google.guava:guava dependency

With the current configuration of the gradle build file, the dependency on the google guava library currently resolves to 20.0-SNAPSHOT
dependencies {
    compile 'com.google.guava:guava:[10.+,)'
    compile 'com.google.code.findbugs:jsr305:2.0.2'

    // junit testing
    testCompile 'junit:junit:4.11'
    testCompile 'org.mockito:mockito-all:1.9.5'
}

Can we freeze the release version of guava-retrying on a release version of guava? Current setup leads to potential instabilities in case there is a breaking change in the edge version of the underlying dependency.
The workaround in my project pom.xml was to exclude the guava dependency as defined
            <dependency>
                <groupId>com.github.rholder</groupId>
                <artifactId>guava-retrying</artifactId>
                <version>2.0.0</version>
                <exclusions>
                    <exclusion>
                        <groupId>com.google.guava</groupId>
                        <artifactId>guava</artifactId>
                    </exclusion>
                </exclusions>
            </dependency>
            <dependency>
                <groupId>com.google.guava</groupId>
                <artifactId>guava</artifactId>
                <version>19.0</version>
            </dependency>

We've had a similar issue. On a larger scale, is guava-retrying using Guava that much to really require Guava? That is, would it make sense to create a utils class with the Guava stuff being used? Guava is a very common library used by projects that uses guava-retrying. One less potential Guava version conflict would be nice.


Add capability to send arbitrary command to Pioneer AVR

Expose aiopioneer.send_command and possibly aiopioneer.send_raw_command as services so that commands not yet implemented can be used. Follow-up from crowbarz/aiopioneer#33.

This has been added in release 0.9.1

Thanks for it! Tried, works, happy :smile:


Allow constructing a ApiClient from a user-supplied Client

stac-async exposes and even documents how to add custom headers to a Client, but falls short of actually being able to use such an instance with ApiClient.
Happy to send a PR if this would be accepted.

Yeah, PRs are very welcome!


7900 XTX Fails to Run

Issue Type
Bug
Tensorflow Version
Tensorflow-rocm v2.11.0-3797-gfe65ef3bbcf 2.11.0
rocm Version
5.4.1
Custom Code
Yes
OS Platform and Distribution
Archlinux: Kernel 6.1.1
Python version
3.10
GPU model and memory
7900 XTX 24GB
Current Behaviour?
I am not entirely sure whether this is an upstream (ROCM) issue, or with Tensorflow-rocm specifically, so I am reporting it to both repo's. A toy example refuses to run and dumps core. I would have expected it to train successfully.
Standalone code to reproduce the issue
import tensorflow as tf
import numpy as np

features = np.random.randn(10000,25)
targets = np.random.randn(10000)

model = tf.keras.Sequential([
     tf.keras.layers.Dense(1)
])

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),
              loss=tf.keras.losses.MeanSquaredError())

model.fit(x=features, y=targets)

Relevant log output
[jaap@Jaap-Desktop code]$ pipenv run python testNN.py
2022-12-24 11:18:37.178811: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
python: /build/hsa-rocr/src/ROCR-Runtime-rocm-5.4.1/src/core/runtime/amd_gpu_agent.cpp:339: void rocr::AMD::GpuAgent::AssembleShader(const char*, AssembleTarget, void*&, size_t&) const: Assertion `code_buf != NULL && "Code buffer allocation failed"' failed.

Are there still people who are waiting for 7900XTX support? Though the performance is still a bit poor, TensorFlow-upstream now runs when built on the latest ROCm release. I was looking into the status of ROCm support for 7900XTX and found a few issues opened by different people and wanted to link all to the issue I opened in MIOpen repo. Though there has not been any confirmation from the developer, I think the performance issues are due to insufficient optimization of MIOpen.
https://github.com/ROCmSoftwarePlatform/MIOpen/issues/2342

I am getting the following error with the latest release (tensorflow-rocm <IP_ADDRESS>0).
2023-12-17 19:48:20.262228: I tensorflow/core/common_runtime/gpu/gpu_device.cc:2015] Ignoring visible gpu device (device: 0, name: , pci bus id: 0000:2d:00.0) with AMDGPU version : gfx1100. The supported AMDGPU versions are gfx1030, gfx900, gfx906, gfx908, gfx90a, gfx940, gfx941, gfx942.

@vampireLibrarianMonk
AFAIK, you need to build TF yourself if you want to use 7900XTX with it. Hopefully, with the release of ROCm6.0, they will release updated frameworks that support 7900XTX out of the box soon. This should give you some idea on how to build it locally if you are interested. To build it on ROCm6, you need to change.

#sed -i 's/5.7.0/5.7.1/g' build_rocm_python3

to

sed -i 's/5.7.0/6.0.0/g' build_rocm_python3

Also, it consumes a lot of memory since it will launch 1 compile process per logical processor you have and each process can consume more than 1GB on average. When I'm building it on my smaller machine (Ryzen 3900X + 32GB), I just disable SMT so that it will only launch 12 concurrent compile processes.

Hi @Mushoz. Has your issue been resolved? If so, please close the ticket. Thanks!

Yes it has, sorry!


Add args for manipulating avro name matches against omero

If the OMERO plate names output by map_series.py are different from the name inside the avro feature files then omerofeatures.py will fail. This PR adds two regular-expression arguments to transform the name in the avro file to that expected in the mapping file.
I originally tried to do this by modifying map_series.py to read the screen file, but the filenames in the screen file are different from the avro feature file names.

Tried --re-pattern "^([A-Za-z0-9_]+).*(\d+)$" --re-match "\1_\2" on the mitocheck mapping you provided (map_screen_1101-dedup.tsv), and it worked as expected.
The reason why names do not match is that the serialize app runs directly on the BF input, in this case on screen files named like LT0001_02--ex2005_11_16--sp2005_02_17--tt17--c3.screen, and by default uses the file's basename (without extension) as the base tag (to which the series number is appended, joined by an underscore). These names are then carried over to the output feature files. This problem can be avoided in the serialize phase, via the -t option, but of course it's not doable after serialization and calculation are done.
Another way to do this without regexes might be to read the screen's *plates.tsv file, which usually has the needed mapping. In any event, the feature added by this PR is an extra tool that's good to have.

Another way to do this without regexes might be to read the screen's *plates.tsv file

That's what I implemented at first:
https://github.com/manics/pydoop-features/commit/f01a8d9ff0c10d78a6652f27be98205ef6f5cdcd
but was unsure whether in the general case it would always be correct to just take the basename and strip the extension. If you think that's a better solution I can open that as a PR instead of this one.

No, just taking the basename from the *plates.tsv file does not work in general. For instance, in the idr0002 case, all input files are called experiment_descriptor.xml, and this is the reason why the -t option was added to the serialize app (it allowed to tag files by plate name instead). In principle (though unlikely to happen in practice), because of the -t option in serialize, the avro names could be totally unrelated to the original input paths.
If we want to go for a two-column mapping file, it should probably be the final mapping needed by omerofeatures.py (e.g., LT0001_02 in the first col and LT0001_02--ex2005_11_16--sp2005_02_17--tt17--c3 in the second). This file would be likely related to the *plates.tsv file, but the exact relationship would depend on the specific screen. This mapping file might have to be created using regexes just like the ones used here though, so it's probably better to just merge this PR for now. We might also add a mapping file option to omerofeatures.py as an extra feature in the future if we want.

Merging for 0.2.0 (i.e. significant bump before we get to 1.0)

+1 for merging


API Fix with working demo

Working of API Fix with a working demo.

Not ready for distribution yet. Some cleanup is necessary.


chore(misc): tiny improvements

Preface: Please let me know if I need a better title.
Description
Improvement 1: I alphabetically sorted some lists in README.MD.
Improvement 2: I added better handling for when the user enters an invalid store name. If they enter an invalid name, it'll print a warning instead of throwing an exception, and if they don't enter any valid names, the app will quit right away.
Improvement 3: To stop more issues like #116 popping up, I added some descriptive logging around the big block of "INFO: Could not find files for the given pattern(s)." output.
Discord: sergo#8847
Testing

Tried with STORES="nvidia"
Tried with STORES="invalid-store"

Looks good to me!