[PyTorch] Fix distributed testing

Description
Adds fixes for PyTorch distributed tests.
Type of change

[ ] Documentation change (change only to the documentation, either a fix or a new content)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Infra/Build change
[ ] Code refractor
[x] Testing

Changes

Install prettytable since it's needed for printing logs from distributed tests.
Change megatron-LM to latest commit which includes fixes to correctly check package version.
Remove sed hack since megatron-LM has removed the deprecated argument apply_query_key_layer_scaling to TE's TransformerLayer.
Omit installing older version of setuptools needed to still import packaging from pkg_resources.

Checklist:

[x] I have read and followed the contributing guidelines
[x] The functionality is complete
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

/te-ci pytorch


Context messages

Adds "Open Documentation" links, diagnostic messages, and corrections in the analysis panel

Diagnostic messages often show line numbers, so #317 would be nice to have soon.
depends on https://github.com/dart-lang/dart-services/pull/663
fixes https://github.com/dart-lang/dart-pad/issues/1676

cc: @bwilkerson @devoncarew

This is super exciting to see. Thank you!

This looks great! Some thoughts on the presentation:

I'd try having the message, correction, and context all be full sentences. This might be as simple as removing the _stripPeriod() and endsWith('.') logic.
You might try having the message and any correction be part of the same span (wrap as one bit of text) rather than be separate divs
perhaps wrap the documentation url and the context message positions? So the doc url would be in the part of the error message w/ the slightly different backgroun

I'd try having the message, correction, and context all be full sentences.

I agree. We actually don't guarantee that we won't write any messages in the future that consist of multiple sentences, in which case removing the period from the last sentence would look odd.

perhaps wrap the documentation url and the context message positions? So the doc url would be in the part of the error message w/ the slightly different background

Wow. I couldn't even tell that the background was different. (Dark mode doesn't work well for my eyes. :-/)
It seems kind of odd to me that the background is different, as if the context messages weren't part of the description of the diagnostic. I think of them as being details that further explain the issue. I'd strongly prefer that we didn't imply otherwise to users.
Also, I noticed something earlier and decided not to mention it because I couldn't think of a concrete example where the situation would arise, but I've just thought of one. What I noticed is that you're displaying the line number for the context message, but the context message is allowed to point to a location in a different file than the one with which the diagnostic is associated, so displaying only the line number in those cases will be confusing.
That situation can't happen using the example you're showing above, but we also create context messages when the issue involves types that have the same name but are actually different. Situations like those cause diagnostics that read something like "The argument type List<int> can't be assigned to a parameter of type List<int>.", which is confusing unless you realize that the two types are from different libraries. So we added text in the message (for clients that can't handle context messages) and added context messages that link to the declarations of the types. To see this in action, try
main() {
  f([0, 1, 2]);
}

void f(List<int> counts) {}

class List<E> {}

Note that this causes the full file paths to be displayed, which is kind of ugly in this context because the user doesn't control the file paths, and will likely lead to confusing context message line numbers.
I suspect that the probability of cases like this will increase when we start supporting package imports.

What I noticed is that you're displaying the line number for the context message, but the context message is allowed to point to a location in a different file than the one with which the diagnostic is associated ...

This may be less of an issue for DartPad - there's only ever one dart file (and, the user doesn't ever see or control that file name).

This may be less of an issue for DartPad - there's only ever one dart file (and, the user doesn't ever see or control that file name).

That's what I thought too and why I didn't mention it before. But paste my example into dartpad and you'll see the file paths displayed today, and in the future one of the context messages will have a line number in core.dart.

I'd try having the message, correction, and context all be full sentences. This might be as simple as removing the _stripPeriod() and endsWith('.') logic.

Done 👍

You might try having the message and any correction be part of the same span (wrap as one bit of text) rather than be separate divs

I tested that out, but felt that it was easier to scan the error message when it was on it's own line. Happy to change it back if we want to try it that way.

perhaps swap the documentation url and the context message positions? So the doc url would be in the part of the error message w/ the slightly different background

The screenshot is misleading, the darker background only shows when you hover over it. Clicking on the error element changes your selection to the error, and clicking on the context message changes your selection to the location associated with that DiagnosticMessage object.
The documentation url is already part of the error message, but before the context messages, so I could move it below the context messages if that's what you mean?

Dark mode doesn't work well for my eyes

We've also gotten some feedback on this (we recently changed the editor theme to be more accessible https://github.com/dart-lang/dart-pad/pull/1767) but this hasn't been deployed yet. We could also look into adding a light theme (used by the embeds) to the main page too.

What I noticed is that you're displaying the line number for the context message, but the context message is allowed to point to a location in a different file than the one with which the diagnostic is associated ...

Oof, it looks we're showing the absolute paths (from dart-services' point of view.) Right now we just grab these from analysis_server_lib and add them to the protobuf message in dart-services:

The argument type 'List<int> (where List is defined in /Users/ryjohn/.dvm/darts/2.10.5/lib/core/list.dart)' can't be assigned to the parameter type 'List<int> (where List is defined in /var/folders/4p/y54w9nqj0_n6ryqwn7lxqz6800m6cw/T/DartAnalysisWrapperlm562s/main.dart)'. - line 2
Open Documentation
List is defined in /Users/ryjohn/.dvm/darts/2.10.5/lib/core/list.dart
int is defined in /Users/ryjohn/.dvm/darts/2.10.5/lib/core/int.dart
List is defined in /var/folders/4p/y54w9nqj0_n6ryqwn7lxqz6800m6cw/T/DartAnalysisWrapperlm562s/main.dart
int is defined in /Users/ryjohn/.dvm/darts/2.10.5/lib/core/int.dart

Is there a way to tell analysis_server_lib how to display these paths?

Dark mode doesn't work well for my eyes

We've also gotten some feedback on this ...

To be clear, I wasn't actually requesting a change, just explaining why I hadn't noticed the difference in background colors. Based on other comments I'm guessing that that's kind of an alternate way of turning a region into a link, not an attempt to visually separate the two sets of messages, so I'm fine with it the way it is.

Is there a way to tell analysis_server_lib how to display these paths?

What is analysis_server_lib?

List is defined in /Users/ryjohn/.dvm/darts/2.10.5/lib/core/list.dart
int is defined in /Users/ryjohn/.dvm/darts/2.10.5/lib/core/int.dart
List is defined in /var/folders/4p/y54w9nqj0_n6ryqwn7lxqz6800m6cw/T/DartAnalysisWrapperlm562s/main.dart
int is defined in /Users/ryjohn/.dvm/darts/2.10.5/lib/core/int.dart

Is there a way to tell analysis_server_lib how to display these paths?

At a guess, the paths are actually part of the string messages in the contexts. We definitely don't want to display those absolute paths - they'll leak info about the filesystem the backend is running on.
We might have to catch a convert likely file paths (lib/core/int.dart ==> dart:core/int.dart, T/DartAnalysisWrapperlm562s/main.dart ==> main.dart).

analysis_server_lib is a package that dart-services currently uses to talk to the analysis server.
Will take a look at changing these paths before they're sent to the client.

Oh, so probably similar to analysis_server_client.
Then, no, there isn't currently any support for changing the file paths. We could consider adding such support if it's easier than converting the paths on the front end. My guess is that converting them on the front end would be easier, though.
As a side not, I've been proposing for a while that we should remove the file paths from the primary message when clients start showing the context messages in a more first-class way. So thanks again for this work. It's getting us one step closer to being able to clean up the messages!

The paths are getting cleaned up in https://github.com/dart-lang/dart-services/pull/664 now. This is ready for another look.


Select one day, but it shows as +1 day

Hello, I'm having trouble with the date range picker. Every time I pick an endDate It ends up +1 day. If i choose 2023-03-03, it shows as "2023-03-04T02:59:59.000Z" in the console.log.
Since my timezone is GMT-3, I suppose it's converting automaticaly to the timezone this library uses after I pick the date, adding +1 day.
Is there a way to solve this ?
Example: (startDate = 2023-03-01, endDate = 2023-03-03

After putting this line of code in 40 <Text style={{color:'black'}}>{JSON.stringify(range.endDate)}</Text> It shows the wrong date (chosen date +1)

This was made on: https://snack.expo.dev/@fitzwabs/react-native-paper-dates-range-picker?platform=web
I just added a single line of code to display the endDate
Locale is set as 'pt'

It picks the end day of the timezone of the device are you on a emulator?

No, I'm using It on my Android device through Expo Go. But this problem happens both with the snack (web) and with my device.
This problem doesn't happen in the demo :https://www.reactnativepaperdates.com/ , so I was hoping there is some trick to solve it

I somehow solved this problem.
I installed moment.js to my project, put this line of code let endDate = moment(range.endDate).utcOffset('-03:00').format('DD/MM/YYYY HH:mm:ss'); and the day isn't +1 anymore.


Replace SubstitutionList with SubstitutionMap in TF extensions

Replace SubstitutionList-related code paths in graph program extraction, constexpr and autodiff with SubstitutionMap APIs.
Replace SILType::getSwiftRValueType (deprecated) with SILType::getASTType.

@swift-ci please test tensorflow

Please take a look at @slavapestov 's feedback above before merging!

@slavapestov Good catch. I'll fix it soon.
Looks like CI is broken. Any suggestions, @marcrasi?

Looks like it got farther with a clean build. The errors now look like real errors uncovered by fixing the Substitution errors.
swift/stdlib/public/core/FloatingPoint.swift.gyb:1876:3: error: unknown declaration syntax exists in the source

By the way, it would be totally fine to land this despite failing CI, if you want the fixes for other errors to go in different PRs. We only need to make sure that CI passes in the final PR that we believe fixes all the errors.

@marcrasi  This error seems impossible. @inlinable is already available in the current branch.
/home/swiftci/jenkins/workspace/swift-PR-TensorFlow-Linux/swift/stdlib/public/core/FloatingPoint.swift.gyb:1876:3: error: unknown declaration syntax exists in the source
  @inlinable // FIXME(sil-serialize-all)
  ^

I think that the error has something to do with parsing the @differentiable attribute, and the fact that it points at @inlinable is a red herring. The error only appears on funcs with @differentiable attribute, and it goes away if I delete the attribute. Want me to investigate it more?

In that case it's probably failing all @differentiable tests. Let me verify.

I tested @differentiable separately and it seems to work for all cases.

@marcrasi There's some strange breakage in parsing, not just @differentiable. When I removed those attributes, I'm getting #tfop errors from building TensorFlow module.
/Users/rxwei/Development/Swift/swift-tf-source-merge/tensorflow-swift-bindings/RawOpsGenerated.swift:13093:10: error: unknown expression syntax exists in the source
  return #tfop("RefSelect",
         ^
/Users/rxwei/Development/Swift/swift-tf-source-merge/tensorflow-swift-bindings/RawOpsGenerated.swift:13118:10: error: unknown expression syntax exists in the source
  return #tfop("RefSwitch",
         ^

@swift-ci please clean test tensorflow

@swift-ci please clean test tensorflow

Merging for now for further debugging.


something about gRPC performance

I had a simple gRPC server based on Python.
In my code, I have one channel and now I find out channel will control and keep connection alive, I can do nothing with channel. If lots of requests, channel will open more than one connection for me. (looks like that it is)
Now I'm testing the performance, If I create 10 process and 10 thread, each on call gRPC server 100 times, looks like:

if client is closed, server will not stop, it will process the rest requests from the client
looks like there is a queue or something like that
the later request in the queue will be slow (1st will be 0.0001s, maybe the 10000 one will be 2s)
some times there will be ExpirationError or NetworkError

Is there anything wrong? or how could I test the performance? any suggestion on deploy?

Closing as a duplicate of issue 5594.


Failure after ASF updated to <IP_ADDRESS>

2023-08-01 09:35:34|dotnet-5928|ERROR|ASF|InitPlugins() Échec dû à une erreur : BoosterManager, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=null
2023-08-01 09:35:34|dotnet-5928|ERROR|ASF|InitPlugins() System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
Could not load type 'EType' from assembly 'ArchiSteamFarm, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=504f46588cc351a9'.
Could not load type 'EType' from assembly 'ArchiSteamFarm, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=504f46588cc351a9'.
Could not load type 'EType' from assembly 'ArchiSteamFarm, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=504f46588cc351a9'.
Could not load type 'ArchiSteamFarm.Steam.Security.Confirmation' from assembly 'ArchiSteamFarm, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=504f46588cc351a9'.
Could not load type 'ArchiSteamFarm.Steam.Security.Confirmation' from assembly 'ArchiSteamFarm, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=504f46588cc351a9'.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.Assembly.GetTypes()
   at ArchiSteamFarm.Plugins.PluginsCore.InitPlugins()
System.TypeLoadException: Could not load type 'EType' from assembly 'ArchiSteamFarm, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=504f46588cc351a9'.
System.TypeLoadException: Could not load type 'EType' from assembly 'ArchiSteamFarm, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=504f46588cc351a9'.
System.TypeLoadException: Could not load type 'EType' from assembly 'ArchiSteamFarm, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=504f46588cc351a9'.
System.TypeLoadException: Could not load type 'ArchiSteamFarm.Steam.Security.Confirmation' from assembly 'ArchiSteamFarm, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=504f46588cc351a9'.
System.TypeLoadException: Could not load type 'ArchiSteamFarm.Steam.Security.Confirmation' from assembly 'ArchiSteamFarm, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=504f46588cc351a9'.

Hello! I have the same error message.

Thank you! 🥳


Add issue templates

Previous discussion in #1164.

Gonna just merge this since we already discussed in #1164 . Just let me know if anything looks wrong or just edit it in master.


Timeline should offer a time picker in addition to a date picker

Hello,
I found this feature in the dev version of frigate and thought this would be a nice feature for the hass-card too:
https://github.com/blakeblackshear/frigate/pull/5183#issuecomment-1634950671
When you got many events on a day the datepicker does not really help one to not end up in endless scrolling and the timeline view was a bit laggy with many events.
Maybe it's also because of my setup that it's partially laggy:

Frigate docker container on a dedicated server (16 cores, 96GB RAM, 32GB cache SSD RAID1, USB Coral / 1G cache, 8G shm)
The storage is on an iscsi-target
Home Assistant runs on RPI4-4GB

Thanks for the idea! Just to make sure I understand, the idea is to add a time picker, as well as a date picker that is already on the timeline?
I guess I didn't think people would have so many events that this would be necessary, but I guess I was wrong!

Hey I think it would especially help in the clips gallery section:

Simply to get a better overview what happened for example yesterday between 6:30 and 6:45. Now I have to scroll through the whole day to see this.
But to have it at both sections would be probably the best.

Oh! That's pretty different than I was imagining thanks for clarifying. So the primary use is in the gallery filter, and in your
example when you use the "Yesterday" time selector, there's just too much scrolling / imprecision?

Yes exactly!

This is now implemented and will be in the next release.


Deduped counts of UTW and UTA documents across multiple books

For example, if a team chooses to do Titus first and then Luke after that, how many articles from Translation Words and how many modules from Translation Academy would be a part of the Luke package, given that they've already translated a handful in the Titus package?

For utw and uta, the references are deduped within a book in order to not overcount due to mulitple references to the same document.
However for the rollup numbers, the same problem must be addressed. My plan is to alter the data provided by the children components to include the document name or uri. Then the parent must first dedup the list before computing their contribution to the total word count.


Component page on doc site should link to the component's type definition file

Type definitions on the documentation site are mechanically generated and do not spark joy. Developers often need to open up a component's type file anyway to better read it. It may be useful to directly link to the source in addition to displaying the types in the website.

Not sure how hard it would be, but it would be awesome if the type in the docs linked to the specific line in the source where the type is defined. Clicking union here for example:


Sniffles joint calling

Hello Khi,
I hope you are well.
I am a postdoc with Sean at umccr. We had a sequencing and analysis done with you in 2023. I wanted to follow up with you about sniffles.
The joint-calling of sniffles appeared to be run per individual, not per cohort. I am wondering if you found jointing calling on normal and tumour tissues more beneficial than the entire cohort.
Many thanks,
Min

Hi Khi,
May I also ask about the naming conventions of the sequencing files, please?
We received a uBAM file named <individual_id>.normal/m84039_230813_021121_s2.hifi_reads.default.bam. What does m84039_230813_021121_s2 denote?
Many thanks,
Min

Hi Min, good to hear from you! Feel free to reach out to me directly via my email at<EMAIL_ADDRESS>too, as we have a close collaboration and I might respond faster there.
To your question: We no longer use Sniffles in the latest pipeline. The current version uses Severus and calls tumor vs normal. On the question of using a full cohort: It could be useful to remove common germline SVs if you have a large enough cohort. The current callset is already filtered against the HPRC healthy cohort (~100 samples) so a lot of the common germline SVs are already filtered out. I would imagine however there's potentially some local ethnicities-specific SVs that are not in the HPRC cohort (which is small by any population seq standard, but useful nonetheless) that remains in there, and a cohort-based joint calling approach can be useful.


add WithPackager to buildpackstore

Summary

This will allow users to create a BuildpackStore that knows how to fetch and package non-packit buildpacks. Users define their own packager that implements the freezer.Packager interface, and BuildpackStore handles the rest. Presto!
Use Cases

Checklist

[ ] I have viewed, signed, and submitted the Contributor License Agreement.
[ ] I have linked issue(s) that this PR should close using keywords or the Github UI (See docs)
[ ] I have added an integration test, if necessary.
[ ] I have reviewed the styleguide for guidance on my code quality.
[ ] I'm happy with the commit history on this PR (I have rebased/squashed as needed).

Blocked on https://github.com/ForestEckhardt/freezer/pull/7

I am fine with this addition. Are you planning on making a create-package packager object as part of occam or will individual buildpack need to implement their own create-package packagers?

@ForestEckhardt I'm not planning to add a packager object to occam, for now. Perhaps in a future refactor we could move the jam packager logic out of freezer and into occam.


PessimisticLockManager concurrency related NullPointerException

The PessimisticLockManager generates occasional NullPointerExceptions when attempting to obtain a lock for an AggregateRoot.  This seems to be triggered when there are multiple threads attempting to obtain the same lock concurrently, although it is sporadic enough to look garbage collection related.
Will attach a test case.

Apologies, not sure how to attach a test case so have pasted it below:
package org.axonframework.repository;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.util.UUID;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import org.axonframework.domain.AggregateIdentifier;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
public class LockManagerTest {
private static final int THREAD_COUNT = 4;
private static final int ATTEMPTS = 3000;
private static final long DELAY_MILLIS = 0;
private LockManager lockManager;
private AggregateIdentifierImpl aggregateIdentifier;

@Before
public void setup() {
	lockManager = new PessimisticLockManager();
	aggregateIdentifier = new AggregateIdentifierImpl();
}

@Test
public void testObtainLock() {
	ExecutorService service = Executors.newFixedThreadPool(THREAD_COUNT);
	LockUnlock[] attempts = new LockUnlock[ATTEMPTS];
    for (int t = 0; t < ATTEMPTS; t++) {
    	attempts[t] = new LockUnlock(t, DELAY_MILLIS);
    }

    long startTime = System.currentTimeMillis();

    for (LockUnlock attempt : attempts) {
    	service.submit(attempt);
    }

    service.shutdown();
    try {
        service.awaitTermination(Long.MAX_VALUE, TimeUnit.SECONDS);
    }
    catch (InterruptedException e) {
    	Assert.fail("Interrupted");
    }

    int failedAttempts = 0;
    for (LockUnlock attempt : attempts) {
    	if (!attempt.success) {
    		failedAttempts++;
    	}
    }
	Assert.assertEquals("Failed LockUnlock count", 0, failedAttempts);
	System.out.println("Completed in " + (System.currentTimeMillis() - startTime) + "ms");
	System.out.println("Minimum expected was " + (ATTEMPTS * DELAY_MILLIS) + "ms");

}

private class LockUnlock implements Runnable {
	private int instanceIndex;
	private long delayMillis;
	private boolean success;

	public LockUnlock(int instanceIndex, long delayMillis) {
		this.instanceIndex = instanceIndex;
		this.delayMillis = delayMillis;
	}
    @Override
    public void run() {
    	int locksAquired = 0;
    	int locksReleased = 0;
    	try {
        	lockManager.obtainLock(aggregateIdentifier);
        	locksAquired++;
        	if (delayMillis > 0) {
            	Thread.sleep(delayMillis);
        	}
        	lockManager.releaseLock(aggregateIdentifier);
        	locksReleased++;

        	success = true;
    	}
    	catch (Exception e) {
    		StringWriter sw = new StringWriter();
    		sw.append("Failed " + instanceIndex + " aquired=" + locksAquired + " release=" + locksReleased + " Exception:");
    		PrintWriter writer = new PrintWriter(sw);
    		e.printStackTrace(writer);
    		System.out.println(sw.toString());
    		Assert.fail(sw.toString());
    	}
    }
}

private static class AggregateIdentifierImpl implements AggregateIdentifier {
	private UUID uuid = UUID.randomUUID();

	@Override
	public String asString() {
		return uuid.toString();
	}

}

}

Extra info: This problem was encountered with both Axon 1.2 & 1.3.1
The error only occurs VERY occasionally in our application, but the TestCase is contrived to throw a LOT of activity at the same AggregateRoot in order to expose the problem more consistently.

Have been given permission to provide the following update:  The code below is a revised version of the PessimisticLockManager.  It passes the above test case consistently, but has just been knocked together quickly so requires further testing to ensure it doesn't break something else.
package org.axonframework.repository;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.locks.ReentrantLock;
import org.axonframework.domain.AggregateIdentifier;
import org.axonframework.domain.AggregateRoot;
import org.axonframework.util.Assert;
/**

This LockManager implementation is based on the Axon PessimisticLockManager, but

fixes a couple of bugs found during concurrency testing.

Needs further testing.
*/
public class RevisedPessimisticLockManager implements LockManager {
private final ConcurrentHashMap<String, DisposableLock> locks = new ConcurrentHashMap<String, DisposableLock>();
/**

{@inheritDoc}
*/
@Override
public boolean validateLock(AggregateRoot aggregate) {
DisposableLock lock = lockFor(aggregate.getIdentifier());
return lock != null && lock.isHeldByCurrentThread();
}

/**

Obtain a lock for an aggregate. This method will block until a lock was successfully obtained.

@param aggregateIdentifier the identifier of the aggregate to obtains a lock for.
*/
@Override
public void obtainLock(AggregateIdentifier aggregateIdentifier) {
boolean lockObtained = false;
while (!lockObtained) {
DisposableLock lock = getOrCreateLock(aggregateIdentifier);
lockObtained = lock.lock();
if (!lockObtained) {
locks.remove(aggregateIdentifier.asString(), lock);
}
}
}

/**

Release the lock held on the aggregate. If no valid lock is held by the current thread, an exception is thrown.

@param aggregateIdentifier the identifier of the aggregate to release the lock for.

@throws IllegalStateException        if no lock was ever obtained for this aggregate

@throws IllegalMonitorStateException if a lock was obtained, but is not currently held by the current thread
*/
@Override
public void releaseLock(AggregateIdentifier aggregateIdentifier) {
DisposableLock lock = lockFor(aggregateIdentifier);
Assert.state(lock != null, "No lock for aggregate was ever obtained");
lock.unlock(aggregateIdentifier);
}

private DisposableLock getOrCreateLock(AggregateIdentifier aggregateIdentifier) {
DisposableLock currentLock = lockFor(aggregateIdentifier);
 if (currentLock == null) {
 	DisposableLock newLock = new DisposableLock();
     currentLock = locks.putIfAbsent(aggregateIdentifier.asString(), newLock);
     return currentLock == null ? newLock : currentLock;
 }

 return currentLock;

}
private DisposableLock lockFor(AggregateIdentifier aggregateIdentifier) {
return locks.get(aggregateIdentifier.asString());
}
private final class DisposableLock {
 private final ReentrantLock lock;
 // guarded by "lock"
 private volatile boolean isClosed = false;

 private DisposableLock() {
     this.lock = new ReentrantLock();
 }

 private boolean isHeldByCurrentThread() {
     return lock.isHeldByCurrentThread();
 }

 private void unlock(AggregateIdentifier aggregateIdentifier) {
     lock.unlock();
     disposeIfUnused(aggregateIdentifier);
 }

 private boolean lock() {
     lock.lock();
     if (isClosed) {
         lock.unlock();
         return false;
     }
     return true;
 }

 private void disposeIfUnused(AggregateIdentifier aggregateIdentifier) {
     if (lock.tryLock()) {
         try {
             if (lock.getHoldCount() == 1) {
                 // we now have a lock. We can shut it down.
                 isClosed = true;
                 locks.remove(aggregateIdentifier.asString(), this);
             }
         } finally {
             lock.unlock();
         }
     }
 }

}
}

Thanks for looking at this quickly.
Probably worth mentioning that line 61 still does this:
            locks.remove(aggregateIdentifier, lock);

But the locks are keyed on aggregateIdentifier.asString() rather than the aggregateIdentifier itself.
Can I ask if/when you would expect to relase an update to the 1.3.1 branch with the fix?
Regards
Nigel

Hi Nigel. Thanks for providing such a good test case. It made it really easy to find and solve the issue.
I usually don't wait too long to do a bug fix release. Since this is a rather annoying one, I'll probably make a release (1.3.2) this weekend.
Cheers,
Allard


Fix compile error detected by Solaris build

Code was introduced when moving triple des to providers..

Checklist

[ ] documentation is added or updated
[ ] tests are added or updated

Thanks Richard. Merged to Master


Add Delegates events on creation/deletion/update

Closes #2299

Added the new event types: NEW_DELEGATE, UPDATED_DELEGATE and DELETED_DELEGATE

I have continued the use of signals to manage the sending of these events to the queue because as we currently have multiple endpoints for delegate management the solution is simpler and clearer. At the moment, the delegates management is not high and there should not be performance issues with the use of signals.


eqwalizer warns about subtypes that can't be possible at the error line

Sorry, if the title is not the best one but I tried to condense everything in a few words.
I  was playing with eqwalizer in a project where I have a module similar to the example below. The module is a gen_server with socket in the #state{}. The socket can be undefined when there is no connection or gen_tcp:socket() when there is one. I have simulated the socket with two atoms: undefined | connected
-module(example).

-behaviour(gen_server).

-export([
    start_link/0,
    init/1,
    handle_cast/2,
    handle_call/3
]).

-record(state, {socket = undefined :: connected | undefined}).

start_link() ->
    gen_server:start_link(?MODULE, [], []).

init([]) ->
    {ok, #state{}}.

handle_call(_Msg, _From, State) ->
    {reply, ok, State}.

handle_cast(_Msg, #state{socket=undefined}) ->
    {noreply, #state{socket=connected}};
handle_cast(_Msg, State=#state{socket=Socket}) ->
  % At this point Socket can only be connected
    send(Socket),
    {noreply, State}.

-spec send(connected) -> ok.
send(connected) ->
    ok.

Output:
error: expected_subtype
   ┌─ apps/example/src/example.erl:28:10
   │
28 │     send(Socket),
   │          ^^^^^^
   │          │
   │          Socket.
Expected: 'connected'
Got     : 'connected' | 'undefined'

   │

  'connected' | 'undefined' is not a subtype of 'connected'
  because
  'undefined' is not a subtype of 'connected'

I was expecting eqwalizer not to raise an error in send(Socket) since at  this point Socket can't be undefined. If it was, the first handle_cast would have match and run the code inside. I can avoid the eqwalizer error by adding when Socket =/= undefined -> in the second handle_cast, but I was wondering if there are plans of supporting this in the future.
Than you

By default eqWAlizer may not refine record fields in general (as it may result if performance issues on average - especially with gigantic records). However, it is possible to force eqWAlizer to be smarter about it in two steps:

Mark the field as refinable by using eqwalizer:refinable(X) marker: (https://github.com/WhatsApp/eqwalizer/blob/9935940d71ef65c7bf7a9dfad77d89c0006c288e/eqwalizer/test_projects/eqwalizer/src/eqwalizer.erl#L47-L54
Add an explicit spec for handle_cast

This works:
...

-record(state, {
  socket = undefined :: eqwalizer:refinable(connected | undefined)
}).

...

-spec handle_cast(term(), #state{}) -> {noreply, #state{}}
handle_cast(_Msg, #state{socket=undefined}) ->
    {noreply, #state{socket=connected}};
handle_cast(_Msg, State=#state{socket=Socket}) ->
    send(Socket),
    {noreply, State}.

Cool! thank you :)


plotly 1.41 features

Bumps plotly.js to v. 1.41.0 and adds all the features included in it.
Addresses #505

So looking at Percy, the orientation attr in the "Stacking" has appeared for a whole bunch of trace types other than scatter/line/area (bar, violin etc) ... can we make this a TraceTypeSection so it doesn't do that plz?

Woo!
Can we get a barpolar percy mock plz?

Blocked by https://github.com/plotly/plotly.js/issues/3013

💃


About the evaluation metrics

Could you please share the code you used to calculate the evaluation metrics in your objective and subjective experiments? I would appreciate it if you did!
Another question is, in your objective experiment, let's say the timbre transfer task, of the 60 music files you generated, is the corresponding timbre in the prompt only piano/viola? And is the corresponding genre only jazz or country? Can it be any other choice? Could you tell me the range of choices for genre and timbre in the prompt? I would appreciate it if you did!

The 60 files are randomly generated. You may find more information in my phd thesis: https://arxiv.org/pdf/2411.12641, where this version I updated some details for MusicMagus.
The principle I choose these pairs is mainly because TransPlayer only supports limited instruments and it is for a fair comparison.
I did not really save the evaluation script. But They are very easy to be implemented with:
https://github.com/LAION-AI/CLAP
https://github.com/gudgud96/frechet-audio-distance
https://librosa.org/doc/0.8.1/index.html


Uncaught LogicException: Could not find supported version of Guzzle

Hello,
I have installed the file without composer in wordpress.
function gapi_update_deleted_url($post_id, $post){

    // bail out if this is not an event item
	if ( 'job' !== $post->post_type ) {
		return;
	}
    //https://github.com/googleapis/google-api-php-client
    //https://developers.google.com/search/apis/indexing-api/v3/prereqs#php
    require_once __DIR__ .'/google-api-php-client-main-2.12.6/google-api-php-client-main/src/Client.php';

    //get deleted url
    $deleted_url = get_permalink($post_id);

    $client = new Google\Client();

    // service_account_file.json is the private key that you created for your service account.
    $client->setAuthConfig( __DIR__.'/axial-history-294207-65cc9c0fd724.json');
    $client->addScope('https://www.googleapis.com/auth/indexing');

    // Get a Guzzle HTTP Client
    $httpClient = $client->authorize();
    $endpoint = 'https://indexing.googleapis.com/v3/urlNotifications:publish';

    // Define contents here. The structure of the content is described in the next step.
    // "URL_UPDATED"
    $content = '{
    "url": "'.$deleted_url.'",
    "type": "URL_DELETED"
    }';
    $response = $httpClient->post($endpoint, [ 'body' => $content ]);
    $status_code = $response->getStatusCode();

    error_log( print_r( $status_code, true ) );

}

Im getting the error "PHP Fatal error: Uncaught LogicException: Could not find supported version of Guzzle. in /var/web/site/public_html/wp-content/plugins/nvs-instant-indexing/google-api-php-client-main-2.12.6/google-api-php-client-main/src/Client.php:1211"

It seems like you are not autoloading or including the Guzzle dependencies as expected. To do this, you'll need to do one of the following:

(recommended) install this library using composer
install this library by downloading the release and including the autoloader
install v5, v6, or v7 of guzzlehttp/guzzle separately


scripts: fix PR compare URLs

Compare the GITHUB_USER's (origin) branch to the upstream branch

Summary

When the scripts open a PR url they should compare the user's fork to the upstream branch (not the user's branch on the upstream repository).
Ticket Link

Fixes
Release Note

cc @asraa

Also -- thanks for checking the push :) I realize I didn't even ask you to do that!


CIAM-2346 Searchable Log Format

Why
Cloud RBAC team needs better searchability of our logs. Currently, in Kibana, there's no structured way to search for HTTP-500. If you search for the freetext "500", you also get results where "500" is anywhere in the freetext. Same with latency. We also got paged for latency in DD, and we wanted to search our request logs by latency, but could not.
Ideally, we would use a JSON format (CIAM-735). I explored this, but it seems like a lot of work (we will start taking on at some point this quarter. See "Alternatives" section for more).
We just want a low-hanging-fruit way to unblock ourselves on these searches.
How
Simply extended the log format we have now.
I do worry a little that our log format, since it conforms to some standard, is considered an API for our on-prem users. Hopefully someone can tell me I'm being too paranoid, and it if they are programmatically parsing these log formats they will be fine. I guess I could make it configurable?
Alternatives
Configurable log format, and provide a nice JSON format. I was hopeful I could do this, but it is not fun to escape characters that Jetty might put into our JSON. I think we will have to write (or find in open source) an implementation of RequestLog that knows how to format and escape.JSON strings. More work than I wanted to take on.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@rigelbm should probably look at this too


Software

Which is the best software to update these hosts in Windows?
I tried Hostsman and it has issues with networks that has only ipv4 and no ipv6!

found this https://scripttiger.github.io/scripts/

Great work @ScriptTiger :+1:


v1.14.7 kafka-controller-post-install job errors due to incorrect RBAC rules

Describe the bug
When deploying the post-install yaml, the kafka-controller-post-install job errors with
failed to list deployments: deployments.apps is forbidden: User \"system:serviceaccount:knative-eventing:knative-kafka-controller-post-install\" cannot list resource \"deployments\" in API group \"apps\" at the cluster scope

This prevents the cleanup of old kafka-broker-dispatcher deployments.
The root cause is a missing verb in the ClusterRole rules.
v1.14.7 post-install which doesn't work:
  - apiGroups:
      - "apps"
    resources:
      - "deployments"
    verbs:
      - "delete"
      - "list"

get added to verbs which allows the job to run:
  - apiGroups:
      - "apps"
    resources:
      - "deployments"
    verbs:
      - "get"
      - "delete"
      - "list"

After the fix, the job runs as expected
kafka-controller-post-install-flxck           0/1     Completed                0          19s

Expected behavior
The job should run with defaults provided in the release artifact.
To Reproduce
Use the v1.14.7 post-install job yaml and run on a fresh k8 cluster.
Knative release version
v1.14.7
Additional context

cc @Cali0707

Closing as #4016 is merged (not sure why that didn't autoclose this one), thanks for finding this @mdwhitley

/close


[QUESTION] EnsureIndex on empty collection

Is it OK to call EnsureIndex on empty collection - at database creation time, to register index fields, or there must be at least one data entry in collection?

@abrca It is perfectly fine to call EnsureIndex for a possibly non-existent collection, it will be created if it doesn't exist.

Thank you


[Question/Bug] Assigning torrents to seasons/episodes returns torrent not found

It's probably best to describe what I'm trying to do using exact examples, because I'm facing compound errors/situation.
I'm trying to watch House M.D. The entire 8 seasons have been added to the Kodi library by a search through Elementum->TV Shows->Search and then Add to Library.
When trying to watch the show though by playing through the library,  while it searches a number of providers, it doesnt seem to find the right torrent though it returns a ton of results all with the word 'house'. This could be because "House" is a generic enough word that appears in a lot of shows. Now, the first question though would be, how would i force Elementum to search for "House M.D." because using that term returns the right results. i've already editted the title of the show in the library to "House M.D." (searching "House M.D." on the provider sites returns the correct torrents, so they do exist)
i'm seeing lines like below in the logs, which doesnt seem to use the changed title.

[magnetdl] Using translated 'original' title 'house'

Now, just searching for the torrent through Elementum->Search and using "House M.D." as a search term definitely returns the right torrent list, and is able to play it. Question though is, when i try to assign this torrent to either S2 of the show or the individual episode itelf, i get the error box saying "torrent is not found" when the torrent is clicked on in the file browser. The torrent also appears in Elementum->Torrents History.
Is this a bug, or am I doing this wrong ? I want to be able to just click on the show in the Kodi library and Elementum then does it's thing as it does for every other show or movie.
Your Environment

Version used: 0.1.83
Environment name and version: Kodi 19.1
Operating System and version: Linux osmc 4.9.113-45-osmc #1 SMP PREEMPT Sun Aug 8 03:43:54 UTC 2021 aarch64 GNU/Linux
Link to Kodi debug log: https://paste.kodi.tv/qezijunude.kodi

Just an update, i restarted Kodi (for other reasons than above), and the torrent seemed to have assigned correctly. Playing episodes straight from the Kodi library just spins up elementum on the season torrent correctly as it should now.
Not sure why this works after a restart, without doing anything else. My only guess is the forced library update on restart may have caused this. Would this assumption be correct ?

@dineshalphaque
now i am a bit confused. after restart you are able to assign torrent from "torrents history" to any item (in elementum or kodi library) and it does not say "torrent is not found"?
can you please try on something new (new torrent and new item in kodi)? just to be sure.
since you use memory as storage - your queue should be empty and torrent should not be found. at least i though so. i honestly do not remember if i tested this feature with memory storage, maybe i did and maybe it worked and i just forgot some nuances why it works with memory (despite the info i described above).

okay, please post update if you will have a chance to test it from scratch.


Fix event horizon + fidling with Time

I had to do some changes to our Time stuff. I really am a noob when it comes to this, so someone with more experience should go over this thoroughly

Build failed /cc: @woksin

Please look over the changes again, @einari @smithmx
I had to make some modifications


(Discussion) High CPU usage in DSpace frontend related to Angular Server Side Rendering (SSR)

Describe the bug
This is a discussion ticket related to ongoing issues with high CPU usage related to Angular SSR.  It's a place to gather public notes & recommendations while developers work on solutions that can be brought back into DSpace code and documentation.
Hints / Notes
A few common tips/hints have been helpful:

First, ensure you are aware of the Performance Tuning Documentation.

Make sure you are using "cluster mode" for PM2!
The built-in, memory-based caching of server-side rendered pages seems beneficial for some sites.  But, be aware it's currently not a shared cache across all nodes in the cluster (each node has its own cache).  Because of this, larger sites may not see much benefit as caching large amounts of content requires a lot of memory.
See some discussion on some example settings in this dspace-tech thread.

Consider disabling Angular's inlineCriticalCss, which can improve SSR performance.

A PR exists to add a new inlineCriticalCss config to the frontend: #2067  (This new config is coming in DSpace 8.0 and 7.6.2)
Make sure to set it to "false" to disable it for better performance. See this PR #2944 for examples.

Some sites have noted that, immediately after upgrading, they were swarmed by bots (doing reindexing, also AI related bots, etc.).  Over time, some have found those bots slow down.  But the early days may require actively blocking or slowing down highly active bots.

Known Community Customizations
What follows are a few (known) customizations from DSpace community members to work around or partially solve these CPU issues:

4Science has updated DSpace-CRIS to exclude the Search component from SSR. This search component is not necessary for SEO, so it may make sense to disable SSR for this component.
PCG has customized DSpace to use Redis for caching (instead of built-in, memory cache).  See notes/slides shared in the Developer Meeting on May 16, 2024.

Brainstorms
Random ideas/brainstorms that are unproven at this time:

Is there a way to use Apache "mod_cache" to cache SSR generated pages?

As with any external cache solution, it may have a negative impact on download/view statistics (in Solr Statistics) as hits to the external cache would not send "viewevents" to the REST API (necessary for statistics).  Unclear how to workaround this limitation of external caching solutions.

Mailing List Discussions
These are (known) mailing lists discussions related to this topic:

"High CPU" discussion on dspace-tech: https://groups.google.com/g/dspace-tech/c/t4H3jJPq0Ck
"DSpace 7.6. slow server response" discussion on dspace-tech: https://groups.google.com/g/dspace-tech/c/ovps9f7glp8/
"DSpace 7.6.1 timeouts in production" discussion on dspace-tech: https://groups.google.com/g/dspace-tech/c/TVJv2R5abNw/

How can you help?
DSpace Committers & Developers are striving to improve the performance of DSpace 7+ to minimize its resource requirements. We welcome brainstorms, solutions, shared code, improved docs that aligns with these goals.  DSpace is community built/maintained software (with no centralized development team). The more details community members can share, the more quickly we can locate/design solutions to benefit everyone.

I want to share that we felt the same described behavior when we went live with our first deployed DS7.

Currently, we are still experiencing constant CPU demand when comparing it to the previous version. Mainly on the Angular processes due to crawling or robots activity.
We use Apache mod_proxy in front of the service. We have created some Apache rules to redirect requests to 404 page not found:
/simple-search?query= and  /handle/HANDLE_PREFIX/ID/simple-search?query=, avoiding angular to render the result for that particular broken URLs.
Like:
<Location "/simple-search">
  Order Deny,Allow
  Deny from all
  Allow from <IP_ADDRESS>
  ErrorDocument 404 "Page not found"
  RewriteEngine On
  RewriteRule .* - [R=404,L]
</Location>

<LocationMatch "/handle/(.*)/(.*)/simple-search">
  Order Deny,Allow
  Deny from all
  Allow from <IP_ADDRESS>
  ErrorDocument 404 "Page not found"
  RewriteEngine On
  RewriteRule .* - [R=404,L]
</LocationMatch>

Also, we use Apache mod_evasive, with some strict restrictions to the traffic (we are still fine tunning it) and we intend to try out some of the above suggestions (like the inlineCriticalCss), and tune even more the values of angular cache.

The built-in memory cache isn't even shared across processes on the same server.
The Redis approach is interesting to me because, while Redis is another in-memory cache, it can be shared across processes.

We tried Apache HTTPD mod_cache here and ran into indiscriminate sharing of Shibboleth tickets across distinct users' sessions.  I haven't had time to investigate, so it might be due to unrelated changes that were hastily backed out at the same time that we disabled the external cache.  If the cache was the culprit, I do not know whether a smarter configuration would fix that.

We (Universität Konstanz) also have made some customizations.
We've opted to use varnish (instead of redis/mod_cache, or built-in cache). This has a few advantages for us:

The cache is shared between all instances
We run a multilingual repo and the built-in cache does not distinguish by language
Finer control over what gets cached when
Things like static assets don't even need to touch the node processes most of the time. This is especially useful as long as the frontend is under high load

We've also disabled SSR for logged-in users, becasue they don't need it and it saves resources for us

4Science has updated DSpace-CRIS to exclude the Search component from SSR. This search component is not necessary for SEO, so it may make sense to disable SSR for this component.

We did something similar, but only for the "related publications list" on person entity pages, as that took quite a while to load. We may extend this to all searches in the future though
We extended the robots.txt to disallow additional robots we don't want

We provide documentation that you can find helpful. There is the solution we used to replace Server-Side rendered pages with Redis in the case study on the Jagiellonian University Repository:
Replace the build-in Server-Side Rendered pages with Redis.pdf

A person with some Angular experience (not a complete expert) asked me why we are making so many small requests to the backend instead of making less but larger requests to the backend. One example could be the submission form where we send one request for each section of the form and one request for each vocabulary and even more requests, instead of having one request for all information we need to build up the form.

There was some good discussion in today's Dev Mtg about areas where we should consider "bundling" many smaller REST Requests into a single (or fewer) requests.  They include:

Submission form (as noted by @pnbecker above)
Authorization checks (as noted by @hutattedonmyarm above)
Search facets (as noted by  @hutattedonmyarm above)
Configuration checks (as noted in https://github.com/DSpace/DSpace/issues/9056)

Those in attendance agreed that we should work on efficiency of these requests.  Ideally we should bundle requests better and send fewer requests per page.
That said, this sort of work requires help/volunteers/investigation.   I'll work on creating some tickets for each of these, as I think they are each separate issues which will require often individual solutions.

There's another one: Browing the landing page of the demo and clicking to go to one of the communities also loads the data twice:

The first one is a 302 redirect, but already contains the community object. Then the actual object is requested again.
#458 also loads data twice.
Not only the number of requests, also the size. E.g. clicking on a publication to go to its page requests the whole "owningCollection" object in a separate request. It's unclear what for and if needed it should be embedded in the item instead of a separate request

@tdonohue  @pnbecker

A person with some Angular experience (not a complete expert) asked me why we are making so many small requests to the backend instead of making less but larger requests to the backend. One example could be the submission form where we send one request for each section of the form and one request for each vocabulary and even more requests, instead of having one request for all information we need to build up the form.

Any multiple request done during the submission form are not relevant for this problem since the submission form is not rendered during the SSR
https://github.com/DSpace/dspace-angular/blob/main/src/app/submission/edit/submission-edit.component.ts#L150
Anyway I agree with the fact that we should optimize the number of request to the server when is possible.
I reckon that when we have multiple requests to the same endpoint referring to different dspace objects, we could try to group them in a single one.
Authorizations endpoint is one example where we have a search method which take multiple uuids for the same authorization feature request (https://github.com/4Science/Rest7Contract/blob/main-cris/items.md#findallbyids). This search method should be used where we handle a set of dspace objects like in the search component.
We did something in this regarding on DSpace-CRIS with the item endpoint to avoid to have multiple request to the item endpoint just to request for embedded objects as done in the search for thumbnails or other objects :

This approach is helpful not only for the SSR but also for the search functionality in general because it allows to speed up the loading of search results.
More in general as well as the high number of REST requests, the SSR can introduce overhead, especially for resource-intensive apps that require heavy server-side processing or complex data computations. In such cases, SSR may strain server resources and slow down rendering.
To mitigate these issues, we should profile and optimize resource-intensive code by using a node monitoring tool.

May I suggest that removing redundant requests, and bundling related requests, are important and should be addressed, but that we may need a second ticket for issues that are unrelated to SSR.

Today, I've moved a number of the performance issues described above into separate tickets. They are not directly related to this SSR issue (or only very loosely related), but obviously are performance issues themselves which require fixing:

#3161
#3162
#3163

If there are other obvious areas where DSpace is (inefficiently) making many requests, then I'd recommend others create additional tickets as well.  While these issues are all semi-related, they all may need to be solved separately, as the solutions may not be identical in all cases.

@paulo-graca : I think your comment may (partially) belong over on #3231, which is the ticket created specifically for this task.
As I've clarified on that ticket, the reason I noted that Search results are "not necessary for SEO" is because we already tell crawlers in our robots.txt file to ignore the search page & facets: https://github.com/DSpace/dspace-angular/blob/main/src/robots.txt.ejs#L11  So, it's odd to have this component available in SSR when (well behaving) crawlers are being told not to use it.
That said, I do agree that we may want to make this configurable, so that sites can choose to allow search results via SSR.  Overall the goal of #3231 is to see if there is "quick fix" performance improvement at least until we can determine ways to improve performance of search in general.
I agree with you that performance of search is NOT ideal, but that also seems like a larger redesign / discussion. Any larger redesign would not help the performance of sites on 7.x or 8.x (as it wouldn't be possible to backport). However, a small change to optionally exclude the search.component from SSR could be backported to 7.x/8.x, as it might be considered a performance fix for those platforms.

Thank you @tdonohue. I agree, I'm proposing a big change that, perhaps, we can't add it easily to DS8 and DS7. I've also added a comment to https://github.com/DSpace/dspace-angular/issues/3231, I agree that it should be configurable. My only point is that it matters in terms of SEO to have Search Results in SSR available for crawlers and also to Internet Archives out there.

This week @autavares-dev and me tried to disable SSR in all pages that not in sitemaps, in short terms all pages that not include "/items/" in url. Our infrainstructure have 5 pods for frontend that had a CPU usage of 1.9 cores, and decreased to 500mcores after this change.
This change will not affect crawllers that follow robots.txt and sitemaps

@vitorsilverio : Could you share how you disabled SSR (by linking to code, or providing an example)?   I'm also not entirely clear what you mean by it started at a usage of "1.9 cores" and "decreased to 500mcores"?   Could you clarify further?
Overall, this sounds promising, but we'd want to understand more about what you've found to be successful.  It might be that others could try a similar approach, or even help make it configurable (so that sites can easily turn SSR off for some pages as needed).

Version: dspace-angular 7.6.1
File: server.ts line 236:
/*
 * The callback function to serve server side angular
 */
function ngApp(req, res) {
  // Only use server-side rendering for specific routes that include items or entities
  if (environment.universal.preboot && (req.url.includes('/items/') || req.url.includes('/entities/')) && req.method === 'GET') {
    // Render the page to user via SSR (server side rendering)
    serverSideRender(req, res);
  } else {
    // If preboot is disabled, just serve the client
    console.log('Universal off, serving for direct client-side rendering (CSR)');
    clientSideRender(req, res);
  }
}

CPU usage:

Untill Fri 23 original code, after with the changes above

@tdonohue after contact from Ollie (@ google scholar) we realize that not all endpoints in sitemaps are using SSR so we change to:
From cd238386ef901915e0e7a384a5a1833dd980d3ce Mon Sep 17 00:00:00 2001
From: Alfeu Uzai Tavares<EMAIL_ADDRESS>Date: Fri, 30 Aug 2024 16:57:30 -0300
Subject: [PATCH] Adiciona paths faltantes para filtro de SSR

---
 server.ts   |  3 ++-
 ssrpaths.ts | 10 ++++++++++
 2 files changed, 12 insertions(+), 1 deletion(-)
 create mode 100644 ssrpaths.ts

diff --git a/server.ts b/server.ts
index 4275a906ad..8bf7f51906 100644
--- a/server.ts
+++ b/server.ts
@@ -56,6 +56,7 @@ import { extendEnvironmentWithAppConfig } from './src/config/config.util';
 import { logStartupMessage } from './startup-message';
 import { TOKENITEM } from './src/app/core/auth/models/auth-token-info.model';

+import { SSR_PATHS } from 'ssrpaths';

 /*
  * Set path for the browser application's dist folder
@@ -238,7 +239,7 @@ export function app() {
  */
 function ngApp(req, res) {
   // Only use server-side rendering for specific routes that include items or entities
-  if (environment.universal.preboot && (req.url.includes('/items/') || req.url.includes('/entities/')) && req.method === 'GET') {
+  if (environment.universal.preboot && req.method === 'GET' && SSR_PATHS.some(p => req.url.includes(p))) {
     // Render the page to user via SSR (server side rendering)
     serverSideRender(req, res);
   } else {
diff --git a/ssrpaths.ts b/ssrpaths.ts
new file mode 100644
index<PHONE_NUMBER>..44a36ce376
--- /dev/null
+++ b/ssrpaths.ts
@@ -0,0 +1,10 @@
+/**
+ * Paths to be SSR
+ */
+export const SSR_PATHS = [
+  '/items/',
+  '/entities/',
+  '/collections/',
+  '/communities'
+];
--
GitLab

2 new endpoints are required by Google Scholar
'/bitstream/' and '/bitstreams/'

I think that we are focusing on the wrong part of the optimization. IMO, there is no problem with SSR as it is in Angular. The issue is how CPU-consuming the whole application is. We can specify a few areas that cause this situation:

A lot of selectors with parameters where memoization doesn't work in NgRX
Everything is pushed to the store and subscribed so store is refreshed a lot, and because of the previous points application get unnecessary subscriptions.

This complicated structure sometimes causes multiple unnecessary data refresh with additional mapping, and the script, instead of mapping data from response, is too busy making selections. That causing the CPU to reach maximum values.
And on the SSR we just render a lot of pages at the same time so the CPU is a crucial resource.

@pcg-kk : I think we'd need more information about your analysis.  You may be entirely correct, but it's difficult to understand what you are looking at without linking to examples or providing suggestions for possible improvement.   Maybe you could document your findings in a Google Doc or on the general DSpace Wiki (e.g. under our "Development > Discussion" page.  I think more details would help others to better understand what you are seeing.
Regardless, I do agree that this ticket is not entirely about SSR.  As you can see from the discussion points above, we've spun off other tickets from this one about endpoints/pages which are sending abnormally large numbers of requests.   So, we are aware that the core bug is not necessarily in SSR, but we are also working to find "quick fix" solutions which can help institutions running 7.x and 8.x to "stabilize" their system if they hit CPU issues specifically in the SSR processing.
In any case, I'd encourage you to share more of your findings...that way I can bring them to the attention of other DSpace developers.

@tdonohue please look into almost any request executed by the frontend. When you look into the initiator tab in dev tools there is a list of (in my example) 1410 methods called between on the subscription before the request is sent and... my browser doesn't present more. I still don't know the first method that was executed at the beginning. We execute a lot of requests and each of them passes that crazy path before is executed.

I can understand that in DSpace we try to create a lot of generic methods that can do a lot of things immediately, but I think that we can agree that this approach that each request is executed in this way is a lost of resources. I don't know the decisions at the beginning why we just don't use effects and actions to call methods, and then success/error actions to save results in the store? Instead we process each request through the store on level before it was send, when it's pending and success. That Request is piped multiple times with a lot of checks and subscriptions, combineLatests.


[bug] leader aa on a new vim tab has issue

how to reproduce

open avante sidebar on one tab, don't close it
open a new vim tab
leader aa

hmm seems like sidebar is not initialized correctly on TabOpen. Should be fixed in #65 cc @yetone


Update ruby to 3.1.6

closes #1032
This also fixes #1032 .  As @EricDurante mentioned, I think SimpleCov was picking up all the extra code from the puma gem, which needed to be pulled directly from Github and stored in vendor/cache.  Also, we no longer need to pull puma from GitHub, the recent release contains the bug fix we needed in the master branch.

delayed job pod is not happy: https://argocd.dev.k8s.libraries.psu.edu/applications/argocd/researcher-metadata-ruby-3-1-6?view=tree&resource=

I was able to get the delayed job pod working with this: https://github.com/psu-libraries/researcher-metadata-config/pull/152 .  I don't think this is a great solution though.  It still raises an error:

but it doesn't fail and exit.  If someone wants to try working on this they can.  Otherwise, I can jump back in when I get back. but I don't think this is quite ready for merge.


iroh: 0.16.0 -> 0.17.0

Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

@cameronfyfe FYI

@crimeminister this one was updated by the bot (https://github.com/NixOS/nixpkgs/pull/314456)

@crimeminister this one was updated by the bot (#314456)

Thanks for the heads up.


Stork env vars not working properly when deployed to k8s

As of Quarkus 3.5.0 you can no longer use lowercase & . notation for environment variable keys (see https://github.com/quarkusio/quarkus/wiki/Migration-Guide-3.5#kubernetes-environment-variables). Because of this, we changed our k8s ConfigMaps to use uppercase with underscore. We quickly found out that this broke the Stork service discovery configuration.
This has been fixed in smallrye/smallrye-config#1041 and smallrye/smallrye-config#1036.
We need to wait for this to trickle into a Quarkus version before this gets fixed here.

@aureamunoz / @radcortez could you please let me know once those changes make it into a Quarkus release?

Follow this one: https://github.com/quarkusio/quarkus/pull/36917 :)

This is also related to quarkusio/quarkus#36856


Samples dose not escapeHtml content property

Control(DomNodeBuilder) kind  had allowHtml property on Enyo1, It is disappeared on Enyo 2.
However, every samples(tutorial) of Enyo2 dose not escapeHtml Content property.
For example, Flickr Search sample dose not escape search results, I can insert Html tag to search results.
The below is the steps to reproduce this issue.

Open "http://enyojs.com/samples/flickr/" on Web Browser.
Search 'blockquote'.
Scroll results list.

Uploaded screenshot to the address below.
http://zawakei.web.fc2.com/testcode/work/20120128_01/index.html

HTML escaping isn't part of the core enyo.Control object, but should be part of widgets based on that for future libraries.


[Progress] Progress bars stacked variation

Hello,
It would be nice if we could stack several progress bars inside a single one, just like the stacked variation in Bootstrap. See here: http://getbootstrap.com/components/#progress-stacked
i'm really looking forward for this feature.
Thanks!

Bump

+1 for this

Does anyone have some custom CSS to make this work and hopefully use some of Semantic's colouring?

Seems to be a duplicate of https://github.com/Semantic-Org/Semantic-UI/issues/6160

Multiple Progress Bars are implemented in Fomantic-UI 2.7.3 now 😄
https://fomantic-ui.com/modules/progress.html#multiple-bars


feat: Only import openjd and deadline modules for Cinema 4D adaptor.

What was the problem/requirement? (What/Why)
I observed that while building the Cinema 4D adaptor in Python 3.12, we would break Cinema 4D's python. This is because Cinema 4D's python was using Python 3.11 and it was failing to import modules that were compiled from Python 3.12.
2024/11/07 10:58:24-06:00 ADAPTOR_OUTPUT: STDOUT: (null)    import win32file
2024/11/07 10:58:24-06:00 ADAPTOR_OUTPUT: STDOUT: (null)ImportError: Module use of python312.dll conflicts with this version of Python.

What was the solution? (How)
Credits to @mwiebe  for helping out with the fix. We essentially only import deadline and openjd modules that the adaptor needs.
This is similar to how Maya does it: https://github.com/aws-deadline/deadline-cloud-for-maya/blob/mainline/src/deadline/maya_adaptor/MayaAdaptor/adaptor.py#L373-L383
What is the impact of this change?
Now the adaptor and Cinema 4D's python versions can be different but it would still work.
How was this change tested?
To test this was a multi-step process:

First I submitted a conda build that would use Python 3.12 for building the Cinema 4D adaptor recipe with these changes as a patch. Cinema 4D python installs the dependencies (pywin32) that it needs.

Next I submitted a simple job to a farm that used this new conda package and confirmed that everything worked as expected.

2024/11/11 18:04:05-06:00 ADAPTOR_OUTPUT: STDOUT: ALF_PROGRESS 94.5833
2024/11/11 18:04:05-06:00 ADAPTOR_OUTPUT: STDOUT: ALF_PROGRESS 95.4167
2024/11/11 18:04:05-06:00 ADAPTOR_OUTPUT: STDOUT: ALF_PROGRESS 95.8333
2024/11/11 18:04:05-06:00 ADAPTOR_OUTPUT: STDOUT: ALF_PROGRESS 96.6667
2024/11/11 18:04:05-06:00 ADAPTOR_OUTPUT: STDOUT: ALF_PROGRESS 97.0833
2024/11/11 18:04:05-06:00 ADAPTOR_OUTPUT: STDOUT: ALF_PROGRESS 97.5
2024/11/11 18:04:05-06:00 ADAPTOR_OUTPUT: STDOUT: ALF_PROGRESS 97.9167
2024/11/11 18:04:05-06:00 ADAPTOR_OUTPUT: STDOUT: ALF_PROGRESS 98.3333
2024/11/11 18:04:05-06:00 ADAPTOR_OUTPUT: STDOUT: ALF_PROGRESS 98.75
2024/11/11 18:04:05-06:00 ADAPTOR_OUTPUT: STDOUT: ALF_PROGRESS 99.1667
2024/11/11 18:04:05-06:00 ADAPTOR_OUTPUT: STDOUT: ALF_PROGRESS 99.5833
2024/11/11 18:04:06-06:00 ADAPTOR_OUTPUT: STDOUT: Finished Rendering
2024/11/11 18:04:06-06:00 INFO: Done Cinema4DAdaptor main
2024/11/11 18:04:06-06:00 Process pid 4064 exited with code: 0 (unsigned) / 0x0 (hex)

Was this change documented?
Is this a breaking change?

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Is there a change we could make to OJD that would let things work without this hack? If so, let's open an issue in its repo.

Hmm, it might not be very straightforward. Ideally if one is using site packages then it should ideally generally work unless there are differences in the versions for python. I'll see if there is an issue in OJD regardless.


Eidolon:Repraised Compat (Dev to Dev)

It's a clone of #23 in a way, but that one talks only about chilled hearts (and it seems fixed?) but ethereal hearts don't seem to draw at all with this mod.
Is there a way we can co-op to make both hearts more compatible? It's mostly legacy code from the former author with little patches to make it work with current overlays, so you'd be the expert on the topic.
Not sure on how much it's only on your end and how much i can help, but if i can avoid forcing all health stack mod to need explicit compat, as you had to, it would be nice

Unfortunately there is no real solution to adding support for all health stacking mods since it is basically required to override vanilla heart renderer with your own. This means that any mod that tries to follow vanilla's heart rendering convention with separate rows will not work as expected or their renderer will not run at all like is the case with Eidolon and there is no real way to detect if a mod has done this without checking if the mod is present.
The reason why Eidolon hearts do not render at all is that the Pre render event in forge is cancelled, which in turn also makes the Post event and any heart overlays that are registered using RegisterGuiOverlaysEvent not run. It is possible for a health stacking mod to run these events by itself, but that is just going to have a ton of these overlays render in multiple rows when health or absorption overflows since they assume that hearts are rendered in multiple rows.
I have added an API for any kind of "effect-like" heart types where a mod just provides their own "alternating" HeartDrawing instances for health and/or absorption to OverlayHeart and everything else is handled internally by rendering those hearts on top of colored hearts. This is what I have done for the chilled effect from Eidolon as you can see here
https://github.com/Terrails/colorful-hearts/blob/0608541d98edfd92e58f9145ab83580d3950ef88/forge/src/main/java/terrails/colorfulhearts/forge/compat/EidolonRepraisedCompat.java#L48-L50
After rechecking how ethereal hearts are rendered I just realized it is actually pretty simple and can be added by subscribing to the post render event provided by Colorful Hearts. https://github.com/Terrails/colorful-hearts/blob/0608541d98edfd92e58f9145ab83580d3950ef88/api/forge/src/main/java/terrails/colorfulhearts/api/forge/event/ForgeHeartRenderEvent.java#L43
Ethereal hearts seem to render after health and absorption hearts, so just assuming that the highest amount of rows is 2 (with health and absorption both having 10 or more hearts), it should be pretty easy to handle it. I initially assumed that you rendered ethereal hearts between health and absorption which would make this more complicated as I would need to add a way to move absorption hearts, but since they are just a thing that renders above it is pretty straightforward. I'll look into adding a compat on my side since I already added it for chilled hearts and it kind of is my fault since these kinds of mods break custom overlays and most of them don't bother with making them compatible.

Adding support was actually pretty simple as you can see here https://github.com/Terrails/colorful-hearts/commit/485fcf5fddece5861c43e800c74caa6cf8c42a7f. I just do the same thing as you do in Eidolon but with it considering that there's only max 20 hearts for absorption and health.
If you want to, I could remove the compat and let you handle it on your own side since the overlay does run if I do not cancel it. You would just have to check if any heart stacking mod like Colorful Hearts is installed using something like ModList.get().isLoaded(modid) and modify the few values to consider that health and absorption together go only to max 20 hearts.
But as you can probably see in my implementation of ethereal hearts, I did not put in the vanilla offset that decreases vertical spacing between heart rows when there's many so that it saves on vertical space. Since Eidolon only has up to 20 hearts afaik (10 from helmet & leggings and another 10 from chestplate) I saw no need for this since there's only a maximum of 4 rows of hearts with health and absorption.


Please provide help thx! r.keyboard() work in the wrong place - see this example

r.init(visual_automation = True)
um.go_href('https://m.baidu.com/')
r.type('#index-form > div:nth-child(1) > div.con-wrap.new-search-con > div > div.input-wrapper > label','how much')
r.keyboard('[ctrl]a')
r.keyboard('[backspace]')
I want to ues keybord[backspace] to delete the words "how much" i type , but eventually it change the top search box.

r.type('#index-form > div:nth-child(1) > div.con-wrap.new-search-con > div > div.input-wrapper > label','how much')

Think it didn't work because the XPath odes not look right.  That one looks like CSS selector.

Just to add on...

For web-based applications, or automating anything in web pages, we usually try not to use visual automation, unless absolutely necessary.  I've been using TagUI for many years.  99.9% of the time, you can automate any web pages using just XPath.

The main reason for using just xpath is that: you can do other work while your TagUI script is running e.g. check your email in Outlook, work on your Powerpoint, or do calculations in Excel.    The moment you used visual automation, similar to UiPath and most of the other RPA tools, it controls your mouse and keyboard, and you cannot do any other work while the robot is running.

For your case, the script is as simple as the following:
import tagui as r
r.init()

r.url("https://m.baidu.com")
r.type('//input[contains(@id, "kw")]', '[clear]')

# the following just for testing
r.wait(1)
r.type('//input[contains(@id, "kw")]', 'RPA[enter]')

Note:

Do not use visual_automation = True in r.init()
Use [clear] to clear the input filed
Your original xpath is not correct
If you use the Chrome Developer Tool to view the HTML for the input field, it will show:

<input type="search" autocomplete="off" autocorrect="off" maxlength="64" id="index-kw" name="word" class="se-input adjust-input" data-sa="" placeholder="输入搜索词" data-sug="1">

The simplest xpath for the above is: //input[@id="index-kw"]

However, I think baidu.com tries to stop people from using RPA on its website.  So every now and then, the code will change to the following:

<input autocomplete="off" autocorrect="off" maxlength="64" id="kw" name="word" type="search" class="se-input new-search-input" placeholder="输入搜索词" value="tagui" data-sug="1">

Note thatid has changed from "index-kw" to "kw".  And the corresponding xpath will be: //input[@id="kw"]

In order to handle both situations, you can use the following xpath : //input[contains(@id, "kw")]

To enter search string into the field, you can use the code: r.type('//input[contains(@id, "kw")]', 'RPA[enter]')

Please give the above code a try and let me know if it works on your side.
One last note: How well you can automate a web page boils down to how good you are with XPath.  XPath is an industry standard to locate any elements on a webpage.  There are many tutorials out there on the Internet.  Just google for "xpath tutorial".  If you are not too familiar with XPath yet, I strongly recommend that you spend some time to build a strong foundation in XPath.
Once you have a good command of XPath, you will be able to read anything on a web page, type any text, click any button, or select any pulldown menu!

Thanks @ck81 for sharing your solution! Adding on @YuZhouQ, for keyboard() function, you first need to use click() or type() with visual automation to actually make the focus on the text box. If you use click() or type() with XPath, the text box will appear to be active but it isn't actually where the key strokes will be sent if you try to type manually.
Something like:
r.click('browser_icon.png') # to make browser active
r.click('text_box.png') # to set focus on textbox
r.keyboard('your message')

Alternatively, use XPath to avoid having to use keyboard() function and automate more easily. You can try out the SelectorsHub Chrome extension to read the XPath value easily.

@YuZhouQ replied in another thread to the above:

Thank you very much for your detailed response to my question. And I'm sorry for my feedback being too late. As the issue was not resolved at the time, I changed my approach.
After reading your answer, I tried again using your method myself. The program ran very smoothly and successfully cleared after entering characters. More importantly, after reading your answer, I truly realized the beauty of xpath. And the differences between Tagui and other RPAs.
Meanwhile, what I want to ask is Is there any difference in the use of CSS_selector, xpath, and other element recognition methods? Is it necessary to keep the browser open when recognizing other elements besides xpath?
Thank you very much for your detailed answer, which has taught me a lot. I didn't have much understanding of xpath before, and it was only through exposure to rpa that I started to understand its use. I will increase my learning of it.

Meanwhile, what I want to ask is Is there any difference in the use of CSS_selector, xpath, and other element recognition methods?

XPath and CSS are the most popular industry standards.
XPath vs CSS
TL;DR: If you can implement a query with CSS selectors, you can do the same with XPath. The other way round is not always possible.

Xpath allows bidirectional flow which means the traversal can be both ways from parent to child and child to parent as well. CSS allows only one directional flow which means the traversal is from parent to child only.
XPath expressions support multiple methods to find the elements which are difficult to locate due to dynamic attribute values or limited attributes. For example, if you need to locate an element that does not have any unique attribute, then you can use XPath axes methods (refer to below) to establish the parent-child relationship with another unique element. In this way, the required elements can be easily located.
XPath has methods such as Contains(), Starts-with() which is very useful in scraping dynamic attribute value when you know some part of the attribute remains constant.
Xpath allows identification with the help of visible text appearing on screen with the help of text() function. e.g. you can use //b/text() to identify the text content of all b elements. CSS does not have this feature.

Note:XPath Axes Methods
XPath axes methods can traverse the DOM bidirectionally and are extremely useful in locating the elements that don’t have any attributes.
There are many axes methods, the most common ones are:

Following
Preceding
Ancestor
Parent
Child

You can google for: xpath vs css which is better
And you will be able to find many articles that explain the above in details with examples.

Is it necessary to keep the browser open when recognizing other elements besides XPath?

No!  If you use 100% Xpath , then you can minimize the browser window and work on other applications such as Outlook or Excel when the TagUI runs.  (But I usually like to keep the browser window open in another monitor so that I can make sure that TagUi is running ok while I do my other work.)  You can even lock your machine and go for your lunch while TagUI runs.  Try this on other RPA tools.  More than 95% of the RPA tools out there in the market cannot do this.  That's why I like TagUI- truly impressive and powerful when it comes to automating web applications!
Again, let me emphasize one more time, this is only true if you use 100% Xpath.  The moment you use visual automation with r.init(visual_automation = True), you should not touch your keyboard and mouse while TagUI runs.


setGitCommitHashEnvironmentVariable and setDDEnvVariables both set DD_TAGS

Expected Behavior
You can easily set your own tags, git commit information, per function tags, etc. and not be concerned with how envs are merged and the order envs are applied.
Actual Behavior
The setGitCommitHashEnvironmentVariable and setDDEnvVariables both call addEnvironment on the Lambda, and therefore the one used last (e.g. if you use const datadog = new DataDog({ tags: 'foo:bar' }); datadog.setGitCommitHashEnvironmentVariable(...fns, "hash")) will 'win'.
Steps to Reproduce the Problem

const datadog = new DataDog({ tags: 'foo: bar' });
datadog.setGitCommitHashEnvironmentVariable(...fns, "d90adafaf8d4b2d82828d514b3ef7c6e66524d0a"))`

Results in:

Specifications

Datadog Lambda Layer version: 0.8 / v2-0.3.3
Node version: 16

@IvanTopolcic looks like related to https://github.com/DataDog/datadog-cdk-constructs/issues/113?

Ah yes, sorry for the duplicate!

Thanks for the report! Going to close this as it's a duplicate.


feat(bundle): add conf parameter to the generate_resources method

This way the conf-source param of the tasks points to the right location.

My mistake, this change is required to make #52 fully working.

Hello @JenspederM,
Have you had a chance to review it? I believe the tests are failing because of lint being picker that in the previous commit.

Hey @npfp,
I have reviewed, and there is essentially nothing wrong with your or. There is however a problem with CI that I currently don't know how to solve.
I will have to spent a bit of time on that, but can merge this in if it's time-critical? 😊

Maybe you know... so the issue is that basically all tests require access to a Databricks workspace and therefore auth secrets.

Perhaps PRs should mostly run unit tests, and let integration tests be run on a nightly job against the main branch? Some bugs might slip, but the contributor experience could be improved.

Ok thx for the explanation. I'll look into it.

Hey @npfp
I resolved the issues regarding testing. And while I was at it, I concorporated these changes. Hope that's okay :)

That's great, thx a lot!


TypeSpec integration with JSON Schema

Services that conform to industry standard data formats, e.g Health Insights and the FHIR standard, would benefit from TypeSpec support to directly import JSON Schema definitions.
An example of where this support would be really useful is the Health Insights service, which works with data that conforms to the FIHR industry standards, which have published JSON schemas, e.g.
observation.schema.json.html

Different requirements:

@useRef for emitter to generate correct output which we currently support

true import of JSON schema into TypeSpec models for in-depth application

@mikekistler, @allenjzhang
We need to be able to reference FHIR objects as external models.
As for clients, FHIR as its own serializers, so it will be great if emitters can give the option to set (by decoration, custom configuration) specific serializers.

BTW - "@useref for emitter to generate correct output which we currently support"
I tried to use it and got "unknown decorator"

I assume we are talking about Json Schema not OpenAPI here, and yeah useRef is not implemented there but should be. I'm not clear if this meets the "directly import" criteria in the OP though.

it will be great if emitters can give the option to set (by decoration, custom configuration) specific serializers.

Can you give more detail here? I'm not sure I understand the scenario yet.

BTW - "@useref for emitter to generate correct output which we currently support"
I tried to use it and got "unknown decorator"

You must import the @typespec/openapi3 package to use the @useRef decorator.
See this link for more information and a Playground example of using the @useRef decorator to get an external reference.

@mikekistler
Is it possible to import openapi3 and generate OpenAPI version 2 json spec ?

Yes ... for OpenAPI v2 you need to use the @azure-tools/typespec-autorest package.
Here's a playground example

What extension/tooling is generating this error?
I think it is likely that whatever it is assumes that the $ref value is a "URL" -- that will actual resolve in an HTTP request.
I tried this same thing but using a true "URL" and I'm not getting an error:

While this has been an epic to start with, it is now a design discussion on a feature request. Removing Epic label.


Peer to peer code review

Congratulations. Very concise and clean code. Keep it up.

Well done. You used the appropriate semantic tags, you have a good code structure. That's all I can say. Great job!!!!!!!


(long term) alongside the 100% actions, dogfood QQ in a github action on itself.

(Can't be implemented until quiet_quality is functional)

oops, this was resolved by #35, forgot the magic 'resolves' description item


Missing interface's property

Why doesn't the second element of the array have the property pageToken?
https://github.com/googleapis/nodejs-storage/blob/b4dbd73189b9fae4c23d614753670ee562bc717b/src/bucket.ts#L90

This property will be present if you pass { autoPaginate: false } into Storage.getFiles.

Hi @olishevskii
That's probably because you don't  have enough objects for the next page. By default, we return 1000 objects. If you do want to limit the number of objects returned, use maxResults option and then you should be able to see nextPage as well.
let [files, nextPage] = await bucket.getFiles({autoPaginate: false, maxResults: 10});

Hope this helps. I'm closing the issue now. Please feel free to re-open it if there's any other issue.

Hi @bajajneha27. Thank you for your answer.
No, it's not working because there's a problem with the interface

I see.
Well, I'm getting the nextPage populated. Can you help me with a code snippet so that I can try to reproduce this?

@bajajneha27 no need to reproduce, the currently defined type is just a generic object. The issue is that it should be more clearly typed which is what @olishevskii is working on in the linked PR.


Installation is never deleted

How to categorize this issue?

/area robustness
/kind bug
/priority 3
What happened:
Suppose an installation fails, because its blueprint cannot be resolved:
  lastError:
    ...
    message: 'unable to find blueprint blueprint in component descriptor: NotFound'
    operation: InitPrerequisites
    reason: ResolveBlueprint
  observedGeneration: 0
  phase: Failed

I run kubectl delete installation ... for the installation. The installation gets a deletion timestamp, but the landscaper does not delete it. After an hour it is still there.
What you expected to happen:
The installation should be deleted.
I am aware that checks are done before a deletion, and that they might need the blueprint. But in this case the failure of the installation occurred so early that no execution and deploy item was created. Shouldn't it be possible to delete the installation?
Environment:
The current version of the landscaper in the master branch.

Here initPrerequisites returns an error, because the blueprint is not found. Therefore, EnsureDeletion is never reached.

same sort of issue is also reported in https://github.com/gardener/landscaper/issues/309


Recommended Patch for Remote Command Execution Vulnerability

Hi @Awilum
Kindly blacklist extensions all uppercase or a combination of uppercase and lowercase (pHP,Php) also in https://github.com/monstra-cms/monstra/blob/dev/plugins/box/filesmanager/filesmanager.admin.php
Line 19:
public static function main()     {         // Array of forbidden types         $forbidden_types = array('html', 'htm', 'js', 'jsb', 'mhtml', 'mht',                                  'php', 'phtml', 'php3', 'php4', 'php5', 'phps',                                  'shtml', 'jhtml', 'pl', 'py', 'cgi', 'sh', 'ksh', 'bsh', 'c', 'htaccess', 'htpasswd',                                  'exe', 'scr', 'dll', 'msi', 'vbs', 'bat', 'com', 'pif', 'cmd', 'vxd', 'cpl', 'empty');
As you can see in the above code, only the lowercase extensions are blacklisted, however, some of these executables can be executed even with extension names in uppercase or a combination of both uppercase and lowercase.
This issue can be patched by converting the $ext into lowercase
This can be done by replacing the below code from the code on line 113,114 and 115 of the index.view.php in https://github.com/monstra-cms/monstra/blob/8c318cdd033a9152a7fba591b37d6739d0c27fce/plugins/box/filesmanager/views/backend/index.view.php:
<?php if (isset($files_list))
        foreach ($files_list as $file) { $ext = File::ext($file);
?>
      <?php if (! in_array(strtolower($ext), $forbidden_types)) {

     $dimension = '';

Hi @Awilum , I have disclosed the vulnerability Pease check: https://blogs.securiteam.com/index.php/archives/3559


Re-added CHIL engine

Re-added CHIL Engine
Now compiles with openssl 1.1
Tested all custom chil/hwchrk functions using a nFast PCI nC1002P/nC3022P ver 2.18.13 card.
Both embed and hwcrhk "application" types work.

Just FYI, it is too late for this to be part of 1.1.1. Thanks for the submission. But is this just restoring code that was removed before?

I refactored the code to compile with openssl 1.1

We removed it in fee65521e7af40d6fcedf68c515f602bdb11f565 with #3389 because we had no way to support it.  Are you taking on that burden starting with the next release?  Also note that we don't have a CLA.

We hoped that it would be updated, but we have removed it from the OpenSSL main source for good reason, maintainability.  It's much better placed as a separate project, or with the vendor of the corresponding hardware.
engine-corner/engine-chil#1 noted, btw.  And yes, that's where I would expect these changes to happen.


Fix missing teamIdentifier and incorrect OS version for userAgent string on macOS

Problem Analysis (Technical)

No team identifier produced for userAgent string on macOS.
Full macOS version description produced for userAgent on macOS instead of short version string
ProcessInfo.processInfo.operatingSystemVersion output is smth like Version 12.6.2 (Build 21G320)), 12.6.2

Solution (Technical)

Added kSecUseDataProtectionKeychain key to the keychain query to obtain kSecAttrAccessGroup value, which contains the team identifier.
Invoke ProcessInfo.processInfo.operatingSystemVersion API to get all the OS version components and properly construct version string.

Give a try to writing a unit test to make sure we get the team identifier. If it doesn't work, that's OK

Well, I made some E2E testing for this to make sure the team id is now a part of user agent string.


Keep language in logo link

Because we don't use cookies, language detection is a bit fragile.
Currently clicking on the logo loses the selected language, it would be great to keep the ?g=fr parameter in the link.
This needs to be done both in JS and in the Go template.

I attempted this issue with this PR: https://github.com/commonsearch/cosr-front/pull/32
The idea is to return to the "full" layout, clear all search related states and timeouts, and use window.history.pushState to append the language info. Please take a look.


Feature request: Tab completion for /include and /exclude arguments

And /run, though that might be more complicated. Would be nice if we could easily take advantage of terminal bash completion scripts

Going to start working on this, but it's a pretty difficult change (need to completely change how we do completions, and make it so that the client sends a request to the server to get all autocompletions, so they work for vscode and other clients as well)

Closed by: https://github.com/AbanteAI/mentat/pull/385


fix: remove use of graphlib.TypeNode in type context

This simplifies the work to resolve routines for fields of structured types and reduces the overall footprint of the type context.

Codecov Report
Attention: Patch coverage is 90.00000% with 4 lines in your changes missing coverage. Please review.

Project coverage is 96.97%. Comparing base (6bd7023) to head (e4742c0).
Report is 1 commits behind head on main.

Files with missing lines
Patch %
Lines

src/typelib/unmarshals/routines.py
57.14%
2 Missing and 1 partial :warning:

src/typelib/ctx.py
90.00%
0 Missing and 1 partial :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main       #4      +/-   ##
==========================================
+ Coverage   96.92%   96.97%   +0.05%
==========================================
  Files          21       21
  Lines        1494     1486       -8
  Branches      267      264       -3
==========================================
- Hits         1448     1441       -7
  Misses         27       27
+ Partials       19       18       -1

Flag
Coverage Δ

unittests
96.70% <90.00%> (+0.04%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Fix the issue of no reply in a threaded mentioned message

Dear reviewer
Issue Summary
This PR is to fix an issue that the bot ignore following mentioned messages in a threaded conversation
The opened issue is #70, which this PR is targeted.
Fix details
The is_no_mention_thread method defined in slack_ops.py returns the negated value than the correct one. This cause the app deems the message in a thread as a not mentioned one and ignore it. So a negate operation was added to fix it.
Would you please kindly review it?
Thanks.

Hi @maxshine, thanks for sharing the issue. The pattern you reported should be supported, but changing this method's behavior breaks an intenational design of this app, which does not allow involving ChatGPT in the middle of a thread convo (this could be pretty annoying because after that, the bot will reply to every single message in the thread.)
The internal method name looks confusing, so I will refactor the code to suppor your use case later

I mean, when you use this app in a channel, this app requires mentioning the app's bot user to start the conversation thread with ChatGPT. In the thread, you don't need to mention the bot. Modifying the internal method breaks the behavior because the method is used in the app_mention event handler too.

I mean, when you use this app in a channel, this app requires mentioning the app's bot user to start the conversation thread with ChatGPT. In the thread, you don't need to mention the bot. Modifying the internal method breaks the behavior because the method is used in the app_mention event handler too.

Thanks for the clarification. I think I get your point. :)

It seems this not relevant. Let me close this PR. Thank you for your time and efforts on this!


Traefik ingress Example on K0s website is outdated.

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  hosts:
  - role: controller+worker
    noTaints: true
    ssh:
      address: <IP_ADDRESS>
      user: root
      keyPath: ~/.ssh/id_rsa
  - role: controller+worker
    noTaints: true
    ssh:
      address: <IP_ADDRESS>
      user: root
      keyPath: ~/.ssh/id_rsa
  - role: controller+worker
    noTaints: true
    ssh:
      address: <IP_ADDRESS>
      user: root
      keyPath: ~/.ssh/id_rsa
  k0s:
    version: v1.28.4+k0s.0
    dynamicConfig: false
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: ClusterConfig
      metadata:
       name: k0s-cluster
      spec:
        network:
          nodeLocalLoadBalancing:
            enabled: true
            type: EnvoyProxy
        extensions:
          helm:
            repositories:
            - name: traefik
              url: https://traefik.github.io/charts
            - name: bitnami
              url: https://charts.bitnami.com/bitnami
            charts:
            - name: traefik
              chartname: traefik/traefik
              version: "20.5.3"
              namespace: default
            - name: metallb
              chartname: bitnami/metallb
              version: "2.5.4"
              namespace: default
              values: |
                configInline:
                  address-pools:
                  - name: generic-cluster-pool
                    protocol: layer2
                    addresses:
                    - <IP_ADDRESS>-<IP_ADDRESS>

The above yaml uses k0sctl v1.28.4+k0s.0, and follows the Traefik Ingress example on  k0s official [docs] while selecting documentation for v1.28.4+k0s.0 (https://docs.k0sproject.io/v1.28.4+k0s.0/examples/traefik-ingress/).
k0sctl bootstraps the cluster with no error but when running k0s kubectl get all (as per example) traefik does not get an external IP assigned.
service/traefik      LoadBalancer   <IP_ADDRESS>   <pending>     80:30193/TCP,443:31634/TCP   74s
daemonset.apps/metallb is also not listed in pods
The example listed on k0s is using an outdated example from 2020 linked here https://traefik.io/blog/from-zero-to-hero-getting-started-with-k0s-and-traefik/

Please post this over on the K0s board
https://github.com/k0sproject/k0s


adding s3 session function instead of client function

to fix bug with Model.create().

This was tested locally with logging statements while creating new llama-2-7b endpoints to confirm that the behavior of Model.create() now works.