Feature/limit fix: maximum context length limit #580

There is a scenario where, assuming our model supports 4096 tokens, we set _maxModelTokens to 3096 and _maxResponseTokens to 1000. Before we check prompt && !isValidPrompt, we have already assigned a value to the message variable, which means that our context will definitely exceed the _maxModelTokens limit. If _maxModelTokens exceeds by a lot, for example, up to 3880, any subsequent response in the conversation that exceeds 400 tokens will result in a context error. Therefore, when we detect this, we need to remove the first item from the context, which is the second item in the systemMessage array, and calculate whether it still exceeds the limit. We need to recursively remove items until the context length is within our set safe limit, in order to ensure that no context errors occur.
[https://github.com/https://github.com/transitive-bullshit/chatgpt-api/issues/542](Make maximum context length limit )

This project is undergoing a major revamp; closing out old PRs as part of the prep process.
This was a good change, but the chatgpt package is pretty outdated at this point. I recommend that you use the openai package or the openai-fetch package.


Stop multiple instances with stop button

Firstly, thanks for the great plugin!
It would be great if there was a stop button or if the existing stop button could stop all created instances. It seems to only stop one at the moment.
I see that hitting play while instances are running will close all existing ones so maybe this logic has already been implemented for a separate button?

Adding a stop button that closes the main scene and all extra windows works for me:
func _stop_pressed():
	kill_pids()
	var interface = get_editor_interface()
	interface.stop_playing_scene()

(Couldn't find a way to connect it to the normal stop button)

Uing this on MacOS, and when I press the Folder button, I get a strange OS window telling me the application couldn't be opened. I presume this might have to do with privileges and/or certificates on MacOS? I'm clueless as to the cause.

the same here


Modularize and cleanup Dashboard page

As you can see in the dashboard page here, its a mess. all the code is inside this single file.
You need to cleanup the code and modularize it, breaking into the following components:
Component Name | File Name

QuickActions - quick-actions.tsx
TasksComponent - tasks.tsx
RecentActivity - recent-activity.tsx

All the component files will be in the /src/components/dashboard folder only. Since these components will require shared states, such as recent activity will be required to be shared by other two components, you will have to be clever with the declaration of sharing of states.
Its a bit challenging issue for someone new to react and next. So only opt for it if you think you can. Furthermore please dont hesitate to ask any doubts you might have

@ChiragAgg5k, could you assign me to this issue? Thanks!

@abhi9ab you are assigned


PageManager PDF Writer deadlocks PrintServices

Steps to reproduce

Install PageManager 9.5 for Windows (Trial version is available)
Reboot
Execute the following code:

PrintService[] printServices = PrinterJob.lookupPrintServices();
for(PrintService printService : printServices) {
    if (printService.getName().equals("PageManager PDF Writer")) {
        System.out.println("Found " + printService);
        printService.getDefaultAttributeValue(PrinterResolution.class);
        System.out.println("Found default attribute value"); // <----- NEVER GETS HERE
    }
}

Platform and architecture:

Windows 10 x64
AdoptOpenJDK HotSpot  11.0.6 2020-01-14
Not tested with other vendors

Workaround

Modify code to continue in the loop when "PageManager PDF Writer" is found; avoiding the use of this printer.
Warning, users are free to rename this printer, so this workaround is not 100% guaranteed.

Downstream Bug Report

https://github.com/qzind/tray/issues/599

Running the offending code through a debugger, the hang seems to occur in Win32PrintService.java:getCapabilities which I believe calls some native function inside WPrinterJob.cpp.

Note, I've also reached out directly to NewSoft Technology Corporation, the current owners of the Presto! line so that they're aware of the issue.

@tresf Before I spend time on this one: Do you have any news from anyone involved?

No reply from NewSoft, no updates on this issue since original filing.

@tresf Thanks for the great analysis (as usual). I tested with an upstream build of OpenJDK 11.0.7 and 14.0.1 and could verify the behavior. Looking at the C++ code and considering that it works well with other printers, I have doubts that the problem is actually in OpenJDK. Blacklisting printers in OpenJDK does not seem to be a sensible idea, either. Most likely, this problem is in PageManager. Do you agree on closing this issue? Otherwise, what's your suggestion?

Our solution downstream was to blacklist it by printer name, but that's only a partial workaround since the default printer name isn't guaranteed to stay that way.

Blacklisting printers in OpenJDK does not seem to be a sensible idea

Agreed.

Most likely, this problem is in PageManager.

Agreed.

Do you agree on closing this issue? Otherwise, what's your suggestion?

If it's decided to close as wont-fix, I'm OK with that, but I don't have enough information to know if that's what I would recommend.  For example, if it's a bug in how Java loads all print drivers that's simply exposed by Presto!, the issue will eventually come back and guarding against this deadlock will help.
However, if the issue is unrelated to Java code, then the bug should go elsewhere (Presto! or Microsoft perhaps?).  At time of writing this, Presto! never got back to me (not even to acknowledge receiving it).
Did you dig deep enough to see if the C++ code was stuck somewhere that could be safely escaped?  I'm curious where it dies.  Even if it's not fixed, referencing the CPP source might help it down the road if it's resurrected, or if Presto! finally decides to correct it.

The C++ implementation is https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/a12f60a83fc87bbba2e2d5ade17f6241a8942aac/jdk/src/windows/native/sun/windows/WPrinterJob.cpp#L726. https://docs.microsoft.com/en-us/windows/win32/api/wingdi/nf-wingdi-devicecapabilitiesa looks like the corresponding Windows API which handles the details and seems to be synchronous. I do not see what could be done to work around this problem, but this isn‘t my area of expertise.

@aahlenst thanks kindly for linking the code.  I haven't run it through a C++ debugger yet (I'm not a C++ developer so if I need to do this, it will take some time) however I noticed by glancing that the API guards for -1 in some areas, but I'm curious if it falls through in other areas?
Quoting Microsoft:

Return value
If the function succeeds, the return value depends on the setting of the fwCapability parameter. A return value of zero (0) generally indicates that, while the function completed successfully, there was some type of failure, such as a capability that is not supported. For more details, see the descriptions for the fwCapability values.
If the function returns -1, this may mean either that the capability is not supported or there was a general function failure.

So an example like this is guarded:
  int cReturned = ::DeviceCapabilities(printerName, printerPort,
                                         dc_id, NULL, NULL);
  RESTORE_CONTROLWORD
  if (cReturned <= 0) { // ######## GUARDED FOR 0 or -1
      JNU_ReleaseStringPlatformChars(env, printer, printerName);
      JNU_ReleaseStringPlatformChars(env, port, printerPort);
      return NULL;
  }

... however some calls don't use <= 0 | > 0 such as copies as well as duplex.
Duplex worries me the most because it doesn't fallback on JNU_ReleaseStringPlatformChars like the other sanitized calls, but instead starts adding DWORD bitwise operators to what could be -1 or 0 according to the API.
I'm sorry for speculative debugging but I don't want to blame Presto! if the issue is an unchecked Win32 API call.

I rather would expect that code to crash or to give funky results than to hang. But I have zero experience with Windows drivers and the JNI code around that, so 🤷♂.
I do not have the expertise to help and it does not seem like the others have, either. Considering you have an executable test case and additional information, it might be worth a try to ask on an OpenJDK mailing list like jdk-dev. Maybe someone with expertise is inclined to respond.

I rather would expect that code to crash or to give funky results than to hang. But I have zero experience with Windows drivers and the JNI code around that, so 🤷♂.

My experience with JNI and Windows has been limited as well.  In my experience, the crashes occur when symbols or registers are incorrect.  Adding insult to injury, I'm not even sure how to run java.exe through a debugger.  On Mac and Linux I've used the CLI tools which allow me to see the backtrace, but that still requires access to debug symbols.  Assuming no one here is going to do that, I believe the next best way to debug this is to make a standalone executable (such as in Visual Studio) and run the Java 11 code as-is against the driver to catch the hang in the IDE.  This is probably a good candidate since it relies on mostly win32 APIs and a few JDK headers.
In regards to reaching out to other channels (like jdk-dev) or staying here, I think it depends on a few factors:

Is this bug severe enough to escalate at all?

Probably not.  Out of thousands of possible print drivers, Presto! is the only one hanging the JDK.

If it WERE worth escalating, who owns it?

If it were worth escalating, I'd probably take this to a contracted team, leaving it open here.  At least that's how I've done it in the past.

I'd like to clarify that symptomatically, it's 100% a support problem.  The end-user can't use the JDK if this driver is installed and that's going to punish the end-users and IT administrators trying to use these two products together.  Fortunately, I've only run into this combination of Presto! and Java this one time, so perhaps someone with an invested interest in both products can push this through the correct channels at a future date. 🍻


VS Code colors change when use two monitors

VSCode Version: 1.31.1
OS Version: Windows 10

Steps to Reproduce:

Launch VS code

Drag it to the second monitor

Contrast changes every time i drag IDE between monitors. I have the same color profile on each monitor, the contrast changes only in VS Code.
it is appeared after 1.30.x update.
Does this issue occur when all extensions are disabled?: Yes

I have the same issue. The colors in vs code IDE are appearing "purpleish" on the external monitor. I am running Ubuntu 18.10. The colors look correct on the laptop monitor, but when I drag the window over to the external monitor, the blues appear purple. I can clearly make out the difference in colors. This only happens with the VS Code windows and not with other applications.

I discovered that version 1.31 now supports color profiles. So you could check if the color profile is the same for both monitors. To bad the release notes did not mention this.

Thanks @anlupat, after messing around with the color profiles, I learned that this is not VS Code specific and affects other applications too - just that it is more noticeable in VS Code (the blue logo and bottom bar really stand out when the colors shift). There is something off with the way Ubuntu is handling my external monitor. Good to know it is not an issue with VS Code (or at least for me). Thanks once again.

I have the same problem.
VS Code Version: 1.33.0
OS Version: Win 10 x64 10.0.17763
first monitor:

second monitor:

We are building exploration builds that use a much newer version of our UI framework (Electron version 6.0.x). I wonder if this issue reproduces with one of these builds, could you try? Download:

macOS: Download
Linux (DEB): Download
Linux (RPM): Download
Windows: Download


[SPARK-30759][SQL] Initialize cache for foldable patterns in StringRegexExpression

What changes were proposed in this pull request?
In the PR, I propose to fix cache initialization in StringRegexExpression by changing case Literal(value: String, StringType) to case p: Expression if p.foldable
Why are the changes needed?
Actually, the case doesn't work at all because of:

Literals value has type UTF8String
It doesn't work for foldable expressions like in the example:

SELECT '%SystemDrive%\Users\John' _FUNC_ '%SystemDrive%\\Users.*';

Does this PR introduce any user-facing change?
No
How was this patch tested?
By the check outputs of expression examples test from SQLQuerySuite.

Test build #118075 has started for PR 27502 at commit 0986e5c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22840/
Test PASSed.

cc @rxin, @gatorsmile , @cloud-fan .
@MaxGekk . Although the above is my first impression, I'll not be against for backporting this.

Test build #118075 has finished for PR 27502 at commit 0986e5c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118075/
Test PASSed.

@HyukjinKwon FYI, since you are in https://github.com/apache/spark/pull/26875 & https://github.com/apache/spark/pull/27514

Test build #118126 has started for PR 27502 at commit 7bc5d9e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22891/
Test PASSed.

Test build #118126 has finished for PR 27502 at commit 7bc5d9e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118126/
Test FAILed.

retest this please

Test build #118135 has started for PR 27502 at commit 7bc5d9e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22898/
Test PASSed.

Test build #118139 has started for PR 27502 at commit 7bc5d9e.

Test build #118139 has finished for PR 27502 at commit 7bc5d9e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118139/
Test FAILed.

Test build #118135 has finished for PR 27502 at commit 7bc5d9e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118135/
Test PASSed.

Good catch! I'm surprised that this bug is exposed after so many years...
Shall we add a test for this bug?

+1 for @cloud-fan 's suggestion.

@dongjoon-hyun @cloud-fan Here is the test https://github.com/apache/spark/pull/27547


Why do we need ng-packagr?

It has a lot of vulnerabilities and I think the latest version will brake our code as Ionic 3 works with Angular <6.
                     === npm audit security report ===

# Run  npm install --save-dev<EMAIL_ADDRESS>to resolve 36 vulnerabilities
SEMVER WARNING: Recommended action is a potentially breaking change

v4 of ng-packagr will break our code!
This is blocked by #27


copr: vectorize dayofyear

Signed-off-by: AerysNan<EMAIL_ADDRESS>PCP #5751
What have you changed?
Vectorize dayofyear.
What is the type of the changes?
Improvement (a change which is an improvement to an existing feature)
How is the PR tested?
Unit test
Does this PR affect documentation (docs) or should it be mentioned in the release notes?
No.
Does this PR affect tidb-ansible?
No.

@sre-bot /run-integration-copr-test copr-test=pr/62

@sre-bot /run-integration-copr-test copr-test=pr/62

@AerysNan Please handle the conflicts

@sre-bot /run-integration-copr-test copr-test=pr/62

@sre-bot /run-integration-copr-test copr-test=pr/62

@sre-bot /run-integration-copr-test copr-test=pr/62

/merge

/merge


Implement disposable for interactive commands

https://gitter.im/scenerygraphics/SciView?at=5cdc460bda34620ff94868ef
Specifically relevant to interactiveconvexmesh

It's on my list to work on this from the framework side. At the moment, implementing Disposable won't work as desired, but shouldn't be too bad to make it do so.

Unfortunately links into gitter have broken now 😭

@kephale A while back, when we switched to Zulip, I dumped the complete gitter histories of all our projects, IIRC including sciview and scenery. I have been planning to put them online somewhere, but never got around to it. Probably the dumped histories have the hash anchors embedded, but I need to check. I don't think I have the dumped data with me while I'm traveling, but I put a reminder in my to-do list to check once I'm back home.


fails to install on npm5 - No matching version found for<EMAIL_ADDRESS>

npm ERR! notarget No matching version found for<EMAIL_ADDRESS>npm ERR! notarget This is most likely not a problem with npm itself.
npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! notarget a package version that doesn't exist.
npm ERR! notarget
npm ERR! notarget It was specified as a dependency of 'xml-crypto'
npm ERR! notarget

Looks like a regression in npm - tracking here: https://github.com/npm/npm/issues/16524

good job with reporting the issue to npm 👏  🎉 👍


Add npm version badge to README

This is great! Thank you so much for this – and for the pull request.


Documented obtaining the root token

/ucc

/uncc

/cc @JoshVanL

Ping, this is just docs - can someone approve etc? Cheers

AWS CLI aws ssm get-parameter supports --with-decryption option, which uses KMS key under the hood to decrypt your SecureString


getting fav.ico and socket.io updates to the react-hot-server-middleware

I have been logging the calls received in the server-hot-middleware function here
I am getting fav.ico and socket.io requrests coming in here as well:
My function looks like this:
export default ({ clientStats }: { clientStats: any }) => async (req: Request, res: Response, next: any) => {
  const preloadedState: any = {};

  console.log('-----------------------');
  console.log(req.url);
  console.log('-----------------------');

  // TODO: find a better way of filtering websocket hot <reloading></reloading>
  if (some(['sockjs', 'hot-update.json', 'favicon.ico'], part => req.url.indexOf(part) > -1)) {
    next();
    return;
  }

  const store = configureStore({ router: { location: req.url} }, createMemoryHistory());

  const context: any = { store };

  store.dispatch(push(req.url));

  const app = (
    <Provider store={store}>
      <StaticRouter location={req.url} context={context}>
        <Application />
      </StaticRouter>
    </Provider>
  );

  const { js, styles } = flushChunks(clientStats, {
    chunkNames: flushChunkNames()
  });

  const appString = renderToString(app);
  const { title } = Helmet.renderStatic();

  console.dir(store.getState());

  res.status(200).send(`
    <!doctype html>

        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
        <meta httpEquiv="X-UA-Compatible" content="IE=edge" />
        <meta charSet="utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />
        ${styles}
        ${title}

${appString}

        ${js}

`);
};

Is this normal?  Should these not be filtered out?

In development mode, this is normal, since HMR has to be active for SSR as well. You are talking about development mode, right?

absolutely, no hmr in production

If I don't filter out the these requests, I get errors like:

bootstrap:102 Uncaught (in promise) SyntaxError: Unexpected token < in JSON at position 5
at Object.parse ()
at XMLHttpRequest.request.onreadystatechange (bootstrap:102)

and

Failed to load resource: net::ERR_INCOMPLETE_CHUNKED_ENCODING

Are you saying that this is ok?

I actually did not experience these errors myself. Do you have a repository with demo code or does this even happen with a clean checkout of the template?
And o.f.c this does not seem to be ok


Using docker-compose examples results in background process errors.

Description
Setting up a new instance of portus using 2.4 and the default docker-compose.yml (and clair-ssl) results in a setup where teh background process is unable to speak to the registry.
Steps to reproduce

git clone from the portus repo.
Copy the example/compose/ directory to a working directory.
Update .env file to use the FQDN of the machine.
Update the docker-compose file to use 2.4 variants of portus in lieu of HEAD. (2.3 tried as well.)
Copy letsencrypt certs into ./secrtes/
Run a docker-compose up .

The logs continuously spam the following....

registry_1    | time="2018-10-26T20:55:59Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.7.6 http.request.host="<IP_ADDRESS>:5000" http.request.id=fa001326-8290-49a5-8099-85d4832d1edd http.request.method=GET http.request.remoteaddr="<IP_ADDRESS>:45440" http.request.uri="/v2/_catalog?n=100" http.request.useragent=Ruby instance.id=42f9a36c-1757-4a9c-acc8-b0fbf435b679 version=v2.6.2
registry_1    | <IP_ADDRESS> - - [26/Oct/2018:20:55:59 +0000] "GET /v2/_catalog?n=100 HTTP/1.1" 401 145 "" "Ruby"
background_1  | Net::OpenTimeout: connection timed out
background_1  | Exception: Net::OpenTimeout: connection timed out

(IP Address changed.  In actuality it's an EIP bound to an AWS instance.)
Deployment information
Mentioned up above.   One tip I've been giving is to check if the registry is reachable from inside of the portus dashboard container. See if you can telnet registry.ip 5000 and get a Connected result. You'll need to zypper in telnet in portus container.

@vitoravelino Thanks for the reply!
I'll need to verify on localhost and see how that works.  As to the other questions, yes connectivity is fine between background worker and the registry containers.
9f1aa748213c:/ # curl https://hub.ifops.net:5000/v2/_catalog
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"registry","Class":"","Name":"catalog","Action":"*"}]}]}

Now that being said, when I login to the registry container, I'm unable to wget against the Portus token endpoint which would seem to be the problem.
In fact, I'm not able to wget https:// endpoints on the internet, but I can hit http:// endpoints.

So taking a stab at it, I opened up the security groups to allow 80/443 from all and adjusted the URLs to exclude the port.  This seems to have done the trick from what I can see.
The big question would be "Why did the container need to route out and then back in?"   Security groups shouldn't matter for something like this I don't think?

I'd imagine an SSL verification check would fail if routing to the respective containers using a service name, so an FQDN is required, which in turn would default to global resolution.

Did anyone figure this out, im not using AWS and still getting this issue. Its rather frustrating please help! the background_1 exits every time i try to start it since trying to implement the TLS with nginx....

@cmeisinger does this setup work out of aws (e.g.: localhost)?
One tip I've been giving is to check if the registry is reachable from inside of the portus dashboard container. See if you can telnet registry.ip 5000 and get a Connected result. You'll need to zypper in telnet in portus container.

@vitoravelino I can telnet from the front end portus container but the background continues to fail. Any suggestions would be amazingly helpful... spent the last 2 days trying to get this thing working.


Fix omit command in User Guide

Signed-off-by: lonelyCZ<EMAIL_ADDRESS>/kind bug
refer here https://github.com/submariner-io/submariner-website/blob/0dd5fdc54627454cf101d999e46040dfade1cfd3/src/content/operations/usage/_index.en.md?plain=1#L614-L615

Thank you @lonelyCZ!


Add CVUSD

Conejo Valley Unified School District

@gideontong Please provide us with:

the university official website URL, if different from the domain you are submitting
an URL of a page on the official website where an IT-related long-term (>1 year) course is offered by the university
an URL of a page or another proof (.pdf or a screenshot are OK) showing that the university recognizes the domain which you are submitting as an official email domain for the students.

@gideontong You might miss my previous comment. Please reply.

@gideontong I am sorry, this request remains unanswered for a while. I have to close it without merging. Please reopen and comment if you have an update regarding the abovementioned issue. Thank you for the understanding!


Slimmer C++ dependencies?

Hi CoACD!
Yuval here from the MuJoCo dev team. Like many others, we are really impressed by CoACD outputs and are enthusiastically recommending it to our users at every opportunity.
We were delighted to discover that you now have a C++ API. Including optional automatic CoACD pre-processing would really improve our user's lives. However, we then saw that your dependencies include extremely large packages like openvdb and boost. We work very hard to keep MuJoCo's dependencies to the absolute minimum, and indeed all of our dependencies are very small and have no dependencies themselves.
Do you have any plans to reduce your dependency payload?
If yes, please let us know.
We would love to help spread your excellent algorithms to the physics-simulating, open-source world 🙂 🌍

Hi, thank you so much for your interest in our library! I'm glad to hear that you would like to include CoACD in MuJoCo. Currently, almost all large packages are for preprocessing meshes in the wild into 2-manifold ones.
OpenVDB offers a good remeshing algorithm though it indeed has a lot of dependencies.
If you have suggestions for a remeshing library with fewer dependencies, I believe I could replace the current one with it, significantly reducing the dependencies.

Could you define "remeshing" more precisely? I'm guessing you need watertightness? Or do you also need resolution {up/down}-scaling?
Here's a proposal for you, which works well especially if you use the remeshing functionality as-is, without exposing knobs to the user. Testing for watertightness (as opposed to remeshing) is really easy (I can show you how, if you like). Perhaps you could add an option to build CoACD without these dependencies at all (exposed to cmake), in which case it throws an error like  "Mesh %s is not watertight. Please fix your mesh or rebuild CoACD with {autoremeshoption}".
I think this addresses everybody's requirements?

This is a great idea - for example, V-HACD is just a single .h file to include. Eliminating heavy dependencies would increase adoption of this library.

Hi I have added an option to disable 3rd party libraries, you can use it by adding -DWITH_3RD_PARTY_LIBS=OFF to cmake options. The code will exit when encountering a non-manifold input.

@SarahWeiii Thanks for adding the disable option! A follow-up request is that now the tarball or zip in the released tag is no longer the most updated with this change, meaning I have to point to the latest commit when using FetchContent_Declare in cmake.
I'm guessing this is not so big of a change worth releasing another version (but maybe a minor bump?) Is there a way that I can avoid pulling down the entire source repo but still using the 3rd party disable option using cmake?

Added a new release

Update from us, we now have an internal draft of MuJoCo-CoAcd integration, hopefully this actually happen soon 🙂
I'll open a new issue if there are any problems.
PS Is it CoAcd or CoACD?

Thank you for the update! I'm eagerly awaiting the release of the final version. 😊
It is CoACD.


Setup backups for K8s

Our K8s clusters store data in four different places on the DataONE Ceph cluster:

Ceph Storage
K8s Storage
Provisioning
K8s cluster

libvirt-pool
RBD VM image
manual
production and dev

k8s-pool-ec42-*
RBD PV
automatic
production

k8sdev-pool-ec42-*
RBD PV
automatic
dev

cephfs
CephFS PV
manual
production and dev

We currently are backing up the VM images in libvirt-pool. Before moving forward on some other issues, like #1, we should be able to restore a broken cluster from a backup.
Backups:

[x] RBD VM images
[ ] RBD PVs (dev)
[ ] RBD PVs (prod)
[ ] CephFS PVs

After spending some time Googling, it looks like Velero is a frequently recommended, open source, free backup options for K8s, which offers (from their website):

Take backups of your cluster and restore in case of loss.
Migrate cluster resources to other clusters.
Replicate your production cluster to development and testing clusters.
You can back up or restore all objects in your cluster, or you can filter objects by type, namespace, and/or label.
Velero is ideal for the disaster recovery use case, as well as for snapshotting your application state, prior to performing system operations on your cluster, like upgrades.

I would like to try this option instead of backend RBD snapshot mirroring and rsync since it appears to solve the initial problem of backing up and deleting unused pods from the cluster, and has other features which look useful, such as snapshotting the entire K8s state before upgrades, and being able to restore back to that state.

Some additional information:

Velero uses object storage to store backups and associated artifacts. It also optionally integrates with supported block storage systems to snapshot your persistent volumes. Before beginning the installation process, you should identify the object storage provider and optional block storage provider(s) you’ll be using from the list of compatible providers.

For the block storage provider, it looks like we can use the Container Storage Interface (CSI), with a fallback of File System Backup if that doesn't work.
For the object storage provider, we can setup a Ceph Object Gateway on the Anacapa Ceph cluster. This is another project, but will be useful for learning how to setup a Ceph Object Gateway for production use on the DataONE Ceph cluster  later on.

Object storage is now available on the Anacapa Ceph cluster: https://github.nceas.ucsb.edu/NCEAS/Computing/issues/254

I installed Velero on k8s-dev, and after some issues have a partially successful backup:
outin@halt:~/velero/velero-v1.12.0-darwin-amd64$ velero backup describe backup-fsb-6
Name:         backup-fsb-6
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.22.0
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=22

Phase:  PartiallyFailed (run `velero backup logs backup-fsb-6` for more information)

Warnings:
  Velero:     <none>
  Cluster:    <none>
  Namespaces:
    hwitw:    resource: /pods name: /cdstool-job--1-n7n6n error: /backup for volume hwitw-data is skipped: pod is not in the expected status, name=cdstool-job--1-n7n6n, namespace=hwitw, phase=Failed: pod is not running
              resource: /pods name: /cdstool-job--1-q5hj2 error: /backup for volume hwitw-data is skipped: pod is not in the expected status, name=cdstool-job--1-q5hj2, namespace=hwitw, phase=Failed: pod is not running
              resource: /pods name: /cdstool-job--1-sdv4x error: /backup for volume hwitw-data is skipped: pod is not in the expected status, name=cdstool-job--1-sdv4x, namespace=hwitw, phase=Failed: pod is not running
              resource: /pods name: /cdstool-job--1-sqlrk error: /backup for volume hwitw-data is skipped: pod is not in the expected status, name=cdstool-job--1-sqlrk, namespace=hwitw, phase=Failed: pod is not running
              resource: /pods name: /cdstool-job--1-st98r error: /backup for volume hwitw-data is skipped: pod is not in the expected status, name=cdstool-job--1-st98r, namespace=hwitw, phase=Failed: pod is not running
              resource: /pods name: /cdstool-job--1-xqp78 error: /backup for volume hwitw-data is skipped: pod is not in the expected status, name=cdstool-job--1-xqp78, namespace=hwitw, phase=Failed: pod is not running
              resource: /pods name: /cdstool-job--1-xxnrv error: /backup for volume hwitw-data is skipped: pod is not in the expected status, name=cdstool-job--1-xxnrv, namespace=hwitw, phase=Failed: pod is not running
    polder:   resource: /pods name: /setup-gleaner--1-5qvwv error: /backup for volume gleaner-context is skipped: pod is not in the expected status, name=setup-gleaner--1-5qvwv, namespace=polder, phase=Failed: pod is not running
              resource: /pods name: /setup-gleaner--1-8fwt9 error: /backup for volume gleaner-context is skipped: pod is not in the expected status, name=setup-gleaner--1-8fwt9, namespace=polder, phase=Failed: pod is not running
              resource: /pods name: /setup-gleaner--1-bdbzr error: /backup for volume gleaner-context is skipped: pod is not in the expected status, name=setup-gleaner--1-bdbzr, namespace=polder, phase=Failed: pod is not running
              resource: /pods name: /setup-gleaner--1-lp2lx error: /backup for volume gleaner-context is skipped: pod is not in the expected status, name=setup-gleaner--1-lp2lx, namespace=polder, phase=Failed: pod is not running
              resource: /pods name: /setup-gleaner--1-pd9dm error: /backup for volume gleaner-context is skipped: pod is not in the expected status, name=setup-gleaner--1-pd9dm, namespace=polder, phase=Failed: pod is not running
              resource: /pods name: /setup-gleaner--1-swmmc error: /backup for volume gleaner-context is skipped: pod is not in the expected status, name=setup-gleaner--1-swmmc, namespace=polder, phase=Failed: pod is not running
              resource: /pods name: /setup-gleaner--1-vffcn error: /backup for volume gleaner-context is skipped: pod is not in the expected status, name=setup-gleaner--1-vffcn, namespace=polder, phase=Failed: pod is not running

Errors:
  Velero:   name: /hwitw-7c7b669857-lz7vc error: /pod volume backup failed: get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed"
            name: /d1index-idxworker-5fd67f856b-hdbvx error: /pod volume backup failed: get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed"
            name: /metadig-controller-595d76dc6c-j6ht7 error: /pod volume backup failed: get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed"
            name: /dev-gleaner-74bf949f4f-555nm error: /pod volume backup failed: data path backup failed: Failed to run kopia backup: Error when processing data/repositories/polder/storage/pos: ConcatenateObjects is not supported
Error when processing data/repositories/polder/storage/pso: ConcatenateObjects is not supported
             name: /dev-gleaner-74bf949f4f-555nm error: /pod volume backup failed: get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed"
  Cluster:    <none>
  Namespaces: <none>

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2023-09-29 13:28:45 -0700 PDT
Completed:  2023-09-29 14:22:19 -0700 PDT

Expiration:  2023-10-29 13:28:45 -0700 PDT

Total items to be backed up:  2206
Items backed up:              2206

Velero-Native Snapshots: <none included>

kopia Backups (specify --details for more information):
  Completed:  66
  Failed:     5

I started docs at https://github.com/DataONEorg/k8s-cluster/blob/main/admin/backup.md

I ran new backups and the previous errors are gone, however some new pods are reporting errors:
Errors:
  Velero:    name: /hwitw-5d876bf94c-khh26 error: /pod volume backup failed: get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed"
             name: /d1index-idxworker-5fd67f856b-hdbvx error: /pod volume backup failed: get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed"
             name: /metadig-controller-595d76dc6c-j6ht7 error: /pod volume backup failed: get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed"
  Cluster:    <none>
  Namespaces: <none>

I ran two backups, and the same three pods failed in both. When viewing the pods with kubectl they show as Running, for example:
outin@halt:~/velero/velero-v1.12.0-darwin-amd64$ kubectl get pod -n hwitw hwitw-5d876bf94c-khh26 -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE             NOMINATED NODE   READINESS GATES
hwitw-5d876bf94c-khh26   1/1     Running   0          3d17h   <IP_ADDRESS>   k8s-dev-node-1   <none>           <none>

While I don't yet understand why these backups are failing, It's possible these are transient errors, I'm going to try the backup again in a few days to see if they are cleared again.
I also setup a nightly backup schedule:
velero schedule create k8s-dev-daily --schedule="0 1 * * *"

I need to reconfigure Velero K8s backups as the Anacapa Ceph cluster has been shut down. My current plan is to install Minio on host-ucsb-27 at Anacapa to use it to receive backup from Velero (Velero requires backing up to object storage). I started setting up host-ucsb-27 today, and hope to get K8s backups running next week.

After spending the time to enable RBD and CephFS snapshots on the two K8s clusters I setup snapshot-based backups with Velero. I tested these successfully with Velero using the csi-rbd-sc storage class, however, PVs without a storage class (ie any that have been created manually, such as gnis/cephfs-gnis-pvc do not have snapshot support and were not backed up. Adding snapshot support to all manually created PVs seems time consuming and not practical.
Errors:
  Velero:    name: /gnis-74c6f7c6df-8vzf2 message: /Error backing up item error: /error executing custom action (groupResource=persistentvolumeclaims, namespace=gnis, name=cephfs-gnis-pvc): rpc error: code = Unknown desc = Cannot snapshot PVC gnis/cephfs-gnis-pvc, PVC has no storage class.

I switched to Velero FSB backups, which can backup all types of PVs, but don’t have the consistency of snapshots. I have not figured out a way to get Velero to use both types of backups, but that may be an option in the future. With FSB instead of CSI snapshots the backup of the GNIS namespace completed successfully. I updated the docs with both CSI and FSB install commands.
I started a full backup of the k8s-dev cluster and was able to back up the entire k8s-dev cluster except for the following:
Errors:
  Velero:    name: /ekbrooke-elasticsearch-data-1 message: /Error backing up item error: /pod volume backup failed: get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed"
             name: /metacatbrooke-dataone-indexer-845fd4c5f5-cnwc2 message: /Error backing up item error: /pod volume backup failed: error exposing host path for pod volume: error identifying unique volume path on host for volume metacatbrooke-temp-tripledb-volume in pod metacatbrooke-dataone-indexer-845fd4c5f5-cnwc2: expected one matching path: /host_pods/e23f6cc5-acfe-411f-8ea9-69fb2bc075c8/volumes/*/metacatbrooke-temp-tripledb-volume, got 0
             name: /metacatbrooke-dataone-indexer-845fd4c5f5-rzzcw message: /Error backing up item error: /pod volume backup failed: error exposing host path for pod volume: error identifying unique volume path on host for volume metacatbrooke-temp-tripledb-volume in pod metacatbrooke-dataone-indexer-845fd4c5f5-rzzcw: expected one matching path: /host_pods/0bb2f50c-04a0-4a1d-9559-49aa7f73a80e/volumes/*/metacatbrooke-temp-tripledb-volume, got 0
             name: /metacatbrooke-dataone-indexer-845fd4c5f5-vkbcx message: /Error backing up item error: /pod volume backup failed: error exposing host path for pod volume: error identifying unique volume path on host for volume metacatbrooke-temp-tripledb-volume in pod metacatbrooke-dataone-indexer-845fd4c5f5-vkbcx: expected one matching path: /host_pods/204f09ca-7e9f-402d-b0e2-12f5b62257b9/volumes/*/metacatbrooke-temp-tripledb-volume, got 0
             name: /hwitw-67ccd577-rltrp message: /Error backing up item error: /pod volume backup failed: get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed"
             name: /d1index-idxworker-5fd67f856b-hdbvx message: /Error backing up item error: /pod volume backup failed: get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed"
             name: /metadig-controller-5ddff7d9fb-jxx77 message: /Error backing up item error: /pod volume backup failed: get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed"

I started checking on hwitw-67ccd577-rltrp and found OOM killed error messages on the host server. This led me to the docs to discover that Velero default memory config is for 100 GiB of data, and cephfs-hwitw-0 maps to /volumes/hwitw-subvol-group/hwitw-subvol/7cb7d655-7ba9-49d2-8dd6-c83a47ff38a1, which contains about 14 TiB of data.
I increased the default memory limits 8x and restarted the backup for hwitw. It appears to be working, it has not run out of memory quickly like it did during the test runs. I'm going to let it run over the weekend out of curiousity, but will probably end up skipping this volume in Velero, and backing up with rsync instead (if it even needs to be backed up).

Increasing the pod memory settings for velero and its agents appears to have fixed the error with large backups, and I was able to finish a backup of the hwitw namespace, of about 14 TiB:
outin@halt:~/velero$ velero backup get hwitw-backup-5
NAME             STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
hwitw-backup-5   Completed   0        0          2024-02-26 09:22:05 -0800 PST   29d       default            <none>

outin@halt:~$ mc du -r velero/k8s-dev
...
257KiB	12 objects	k8s-dev/backups/hwitw-backup-5
14TiB	710551 objects	k8s-dev/kopia/hwitw
...

I started a second incremental backup run of the entire cluster.

A full backup of K8s-dev completed except for three resources:
    Failed:
      brooke/metacatbrooke-dataone-indexer-845fd4c5f5-cnwc2: metacatbrooke-temp-tripledb-volume
      brooke/metacatbrooke-dataone-indexer-845fd4c5f5-rzzcw: metacatbrooke-temp-tripledb-volume
      brooke/metacatbrooke-dataone-indexer-845fd4c5f5-vkbcx: metacatbrooke-temp-tripledb-volume

Longer error messages are:
Errors:
  Velero:    name: /metacatbrooke-dataone-indexer-845fd4c5f5-cnwc2 message: /Error backing up item error: /pod volume backup failed: error exposing host path for pod volume: error identifying unique volume path on host for volume metacatbrooke-temp-tripledb-volume in pod metacatbrooke-dataone-indexer-845fd4c5f5-cnwc2: expected one matching path: /host_pods/e23f6cc5-acfe-411f-8ea9-69fb2bc075c8/volumes/*/metacatbrooke-temp-tripledb-volume, got 0
             name: /metacatbrooke-dataone-indexer-845fd4c5f5-rzzcw message: /Error backing up item error: /pod volume backup failed: error exposing host path for pod volume: error identifying unique volume path on host for volume metacatbrooke-temp-tripledb-volume in pod metacatbrooke-dataone-indexer-845fd4c5f5-rzzcw: expected one matching path: /host_pods/0bb2f50c-04a0-4a1d-9559-49aa7f73a80e/volumes/*/metacatbrooke-temp-tripledb-volume, got 0
             name: /metacatbrooke-dataone-indexer-845fd4c5f5-vkbcx message: /Error backing up item error: /pod volume backup failed: error exposing host path for pod volume: error identifying unique volume path on host for volume metacatbrooke-temp-tripledb-volume in pod metacatbrooke-dataone-indexer-845fd4c5f5-vkbcx: expected one matching path: /host_pods/204f09ca-7e9f-402d-b0e2-12f5b62257b9/volumes/*/metacatbrooke-temp-tripledb-volume, got 0
  Cluster:    <none>
  Namespaces: <none>

I'm looking into fixing these errors.

It appears that the pod volumes have an issue with their storage:
$ kubectl describe pod metacatbrooke-dataone-indexer-845fd4c5f5-cnwc2 -n brooke

...
  metacatbrooke-temp-tripledb-volume:
    Type:          EphemeralVolume (an inline specification for a volume that gets created and deleted with the pod)
    StorageClass:  csi-cephfs-sc-ephemeral
    Volume:
    Labels:            <none>
    Annotations:       <none>
    Capacity:
    Access Modes:
    VolumeMode:    Filesystem
...

I'll check with @artntek before proceeding...

@nickatnceas - those volumes are ephemeral, basically acting as a short-term local cache - by definition, they are non-critical and will be regenerated as needed. If there's a way of excluding them from the backups, that would be the best bet, I think.

I excluded the three pod volumes from backup and the namespace backup for brooke now completes successfully.
velero backup create brooke-backup-1 --include-namespaces brooke

kubectl -n brooke annotate pod/metacatbrooke-dataone-indexer-845fd4c5f5-cnwc2 backup.velero.io/backup-volumes-excludes=metacatbrooke-temp-tripledb-volume

velero backup create brooke-backup-2 --include-namespaces brooke

kubectl -n brooke annotate pod/metacatbrooke-dataone-indexer-845fd4c5f5-rzzcw backup.velero.io/backup-volumes-excludes=metacatbrooke-temp-tripledb-volume
kubectl -n brooke annotate pod/metacatbrooke-dataone-indexer-845fd4c5f5-vkbcx backup.velero.io/backup-volumes-excludes=metacatbrooke-temp-tripledb-volume

velero backup create brooke-backup-3 --include-namespaces brooke

NAME                STATUS            ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
brooke-backup-1     PartiallyFailed   3        0          2024-02-27 16:11:08 -0800 PST   29d       default            <none>
brooke-backup-2     PartiallyFailed   2        0          2024-02-27 16:15:33 -0800 PST   29d       default            <none>
brooke-backup-3     Completed         0        0          2024-02-27 16:21:39 -0800 PST   29d       default            <none>

I added the exclusion instructions to the backup docs, and started another full namespace backup.

A full backup run reported it completed!
There are a couple of warnings in the backup log, but they appear to be for broken pods, which are probably safe to ignore:
$ velero backup create full-backup-4
Backup request "full-backup-4" submitted successfully.
Run `velero backup describe full-backup-4` or `velero backup logs full-backup-4` for more details.

$ velero get backups
NAME                STATUS            ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SE
full-backup-1       PartiallyFailed   7        0          2024-02-23 11:51:44 -0800 PST   25d       default            <none>
full-backup-2       PartiallyFailed   3        2          2024-02-26 09:41:42 -0800 PST   28d       default            <none>
full-backup-3       PartiallyFailed   3        2          2024-02-27 10:54:16 -0800 PST   29d       default            <none>
full-backup-4       Completed         0        2          2024-02-27 16:24:51 -0800 PST   29d       default            <none>

$ velero backup describe full-backup-4
Name:         full-backup-4
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.22.0
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=22

Phase:  Completed

Warnings:
  Velero:     <none>
  Cluster:    <none>
  Namespaces:
    pdgrun:   resource: /pods name: /parsl-worker-1708968600101 message: /Skip pod volume pdgrun-dev-0 error: /pod is not in the expected status, name=parsl-worker-1708968600101, namespace=pdgrun, phase=Pending: pod is not running
              resource: /pods name: /parsl-worker-1708968600248 message: /Skip pod volume pdgrun-dev-0 error: /pod is not in the expected status, name=parsl-worker-1708968600248, namespace=pdgrun, phase=Pending: pod is not running

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Or label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto
Snapshot Move Data:          false
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2024-02-27 16:24:51 -0800 PST
Completed:  2024-02-27 16:39:35 -0800 PST

Expiration:  2024-03-28 17:24:51 -0700 PDT

Total items to be backed up:  2654
Items backed up:              2654

Backup Volumes:
  Velero-Native Snapshots: <none included>

  CSI Snapshots: <none included>

  Pod Volume Backups - kopia (specify --details for more information):
    Completed:  85

HooksAttempted:  0
HooksFailed:     0

$ kubectl get pods -n pdgrun
NAME                         READY   STATUS             RESTARTS   AGE
parsl-worker-1708968600101   0/1     InvalidImageName   0          31h
parsl-worker-1708968600248   0/1     InvalidImageName   0          31h

I made the following changes:

Set up a nightly backup schedule for k8s-dev
Started backing up k8s-prod
- Created new bucket and user in MinIO at Anacapa
- Installed Velero pods on k8s-prod
- Ran a test backup of the purser namespace
- Started a full backup

I updated the first comment to include new ticket requirements.

Received the following errors after running the full backup of k8s-prod:
Errors:
  Velero:    name: /gnis-6c7f9d9bb7-8mg4j message: /Error backing up item error: /pod volume backup failed: get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed"
             name: /metadig-controller-7db96b7585-zb2dk message: /Error backing up item error: /pod volume backup failed: get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed"
             name: /metadig-solr-0 message: /Error backing up item error: /pod volume backup failed: get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed"
             name: /prod-gleaner-76df9dfc54-rp9x8 message: /Error backing up item error: /pod volume backup failed: get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed"
             name: /prod-gleaner-76df9dfc54-rp9x8 message: /Error backing up item error: /pod volume backup failed: get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed"

    Failed:
      gnis/gnis-6c7f9d9bb7-8mg4j: gnis-volume
      metadig/metadig-controller-7db96b7585-zb2dk: metadig-pv
      metadig/metadig-solr-0: data
      polder/prod-gleaner-76df9dfc54-rp9x8: s3system-volume, triplestore-volume

I was able to fix the failing backup by increasing the memory limits on Velero and its pods to double of what I used on k8s-dev:
kubectl patch daemonset node-agent -n velero --patch '{"spec":{"template":{"spec":{"containers":[{"name": "node-agent", "resources": {"limits":{"cpu": "4", "memory": "16384Mi"}, "requests": {"cpu": "2", "memory": "8192Mi"}}}]}}}}'
kubectl patch deployment velero -n velero --patch '{"spec":{"template":{"spec":{"containers":[{"name": "velero", "resources": {"limits":{"cpu": "4", "memory": "8192Mi"}, "requests": {"cpu": "2", "memory": "2048Mi"}}}]}}}}'

I also set a nightly backup schedule with 90 days of retention, and the first scheduled backup ran successfully last night:
outin@halt:~$ velero schedule get
NAME          STATUS    CREATED                         SCHEDULE    BACKUP TTL   LAST BACKUP   SELECTOR   PAUSED
full-backup   Enabled   2024-03-20 17:12:47 -0700 PDT   0 1 * * *   2160h0m0s    22h ago       <none>     false

outin@halt:~$ velero backup get
NAME                         STATUS            ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
full-backup-20240321010020   Completed         0        33         2024-03-20 18:00:20 -0700 PDT   89d       default            <none>
...

I setup backup monitoring on optimal-squirrel.nceas.ucsb.edu to alert when Velero fails to complete a backup, or if the backups silently stop running, for the nightly backups of k8s-prod and k8s-dev.
The backup scripts are in https://github.nceas.ucsb.edu/outin/check_velero_backups
Check_MK alerts:


Include service name as field in downloaded data

Requesting a feature to include the service name (EC2, EBS, DynamoDB, etc) as a field in the output. This will allow users to aggregate findings together in order to report/remediate tagging discrepancies.

Hi @saxesrule,
Thanks for providing your input.  You would like us to add the service name to the list of found resources & tags for both the browser/UI view & the CSV download.  We will add this to our feature request list & evaluate it.  Best!


tests: small adjustments for integration tests

small fixes suggested by @yorinasub17

small fixes suggested by @yorinasub17


Cannot Full Screen or Copy/Paste in High Sierra Developer Beta 2 when opening through terminal

VSCode Version: Code 1.13.1 (379d2efb5539b09112c793d3d9a413017d736f89, 2017-06-14T18:13:05.928Z)
OS Version: Darwin x64 17.0.0
Extensions:

Extension
Author
Version

beautify
HookyQR
1.1.1

material-icon-theme
PKief
2.0.9

vscode-docker
PeterJausovec
0.0.16

code-settings-sync
Shan
2.8.1

html-css-class-completion
Zignd
1.7.0

vscode-eslint
dbaeumer
1.2.11

githistory
donjayamanne
0.2.0

vscode-babel-coloring
dzannotti
0.0.4

gitlens
eamodio
4.1.3

vscode-npm-script
eg2
0.2.0

git-project-manager
felipecaputo
1.3.2

theme-material-theme
jprestidge
1.0.1

prettify-json
mohsen1
0.0.3

debugger-for-chrome
msjsdiag
3.1.4

vscode-icons
robertohuertasm
7.9.0

sass-indented
robinbentley
1.4.1

language-stylus
sysoev
1.7.3

JavaScriptSnippets
xabikos
1.4.0

ReactSnippets
xabikos
1.3.0

Material-theme
zhuangtongfa
2.8.9

Steps to Reproduce:

Close VSCode (make sure you right click -> Quit in the Dock)
Open VSCode through terminal by running code

I believe this was not an issue in Beta 1. So maybe it will be fixed with the next release of macOS?

@gorjuspixels tmux?

@bpasero yeah only happens when using tmux.. is this a known issue on previous macOS versions?

Read https://medium.com/@joaomoreno/persistent-terminal-sessions-in-vs-code-8fc469ed6b41

@bpasero thanks, I've set up persistent terminal sessions in vs code.. but I don't think that's the problem. it doesn't fix the issue.
when I open VSCode through a tmux session, by running code ., im unable to copy/paste in the editor and unable to enter full screen

@gorjuspixels did you set-option -g default-command "reattach-to-user-namespace -l zsh"?

@bpasero yep. this seems to be a duplicate of #13366

fixed after running brew install reattach-to-user-namespace


Migrate onChange validation to occur onBlur by default for inputs

Ticket tracking issue: https://selectquote.atlassian.net/browse/SSR-149
Overall would like to see inputs validate inputs onBlur so that validation errors/warnings do not throw while the user is typing/making changes.
Loom!  https://loom.com/share/0627d64b2ac347cbb395ac25edd30157

After discussions and thoughts in both our frontend chapter meetings and UX team, I'm deciding to keep the current behavior as default while making validation onBlur and onChange as options that can be set to true or false via new validateOnBlur and validateOnChange props.


Feat: hide edge labels arg

This PR is related to the issue number #1343.
It adds an option that hides the edge labels.

Coverage decreased (-0.05%) to 71.015% when pulling b252ad59c855af3ea2c3ffc51c0229333a2d333c on karranb:karran/feat/hide-edge-labels-arg into cb66b31d287bbd14a08b03b2a311bb96d28ce473 on django-extensions:master.

Coverage decreased (-0.05%) to 71.015% when pulling b252ad59c855af3ea2c3ffc51c0229333a2d333c on karranb:karran/feat/hide-edge-labels-arg into cb66b31d287bbd14a08b03b2a311bb96d28ce473 on django-extensions:master.

Awesome !
Could you add some tests in tests/management/commands/test_graph_models.py to make sure it behaves as expected and to catch regressions later ?
And a few lines in docs/graph_models.rst to document it's existence and purpose :-)

hi @trbs, I added the tests and documentation as requested 😄. The CI build failed, but by looking at the logs, it does not look like it's related to this PR changes to me though.

Awesome thanks !
yeah the build failure was an unrelated (graph_models) change :-)


Add DicomINRDataset

Codecov Report
Attention: Patch coverage is 98.14815% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 93.96%. Comparing base (a27eec6) to head (ef06977).

Files
Patch %
Lines

torch_dicom/datasets/dicom.py
98.11%
1 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##           master      #69      +/-   ##
==========================================
+ Coverage   93.84%   93.96%   +0.11%
==========================================
  Files          22       22
  Lines        1787     1838      +51
==========================================
+ Hits         1677     1727      +50
- Misses        110      111       +1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


 Aside from a few issues here and there, I find seccubus to be pretty neat.
Trying to get Zap scans to work for me have been an issue. Running the scan with vanilla parameters didn't work since seccubus was looking for literally zap.jar and zap had changed the name to zap-2.6.0.jar instead. I believe even specifying the path to that file didn't work because seccubus was still looking for only zap.jar, so I literally just copied the jar file to be zap.jar for seccubus' sake.
After that it was running and I tried to view the results of the scan after it had said it was finished, but the scan results were always empty or not there at all. I decided to run the scan with the -v verbose output flags and saw that there were some errors.
Running zap2ivil
syntax error at line 1, column 0, byte 0 at /usr/lib/x86_64-linux-gnu/perl5/5.24/XML/Parser.pm line 187
XML::Simple called at /opt/seccubus/bin/zap2ivil line 69.
Reading in ZAP xml report file
running load_ivil
File does not exist: at /opt/seccubus/SeccubusV2/SeccubusIVIL.pm line 98
Reading file /tmp/seccubus.[nonce].ivil.xml into memory
file loaded parsing data
The scripts to attatch the 2 files, xml and ivil reports appear to always run correctly without problems
I assume the generated xml report is not formatted or being parsed correctly. At this point I figured if it wasn't running right using the java way to run zap, I might as well try out the python api and see if it generates an xml report that seccubus deems good enough, but I'm met with the same errors.(for that I just changed what was actually executed as a command in the /ZAP/scan file. I almost kinda prefer this way since I enjoy the API a lot more and you can give paths to different types of scans like a basic scan with nothing special that you've made or a scan that creates sessions and runs the scans as authenticated users) Even if I just got the xml file to appear on the web interface it would be beneficial to me.

I've been trying to dive into why this is happening myself, and I found that there was a few non-XML output being written to the /tmp/seccubus.nonce file at the start of the files which was throwing off the parser, hence the syntax error at line 1. The only trouble I'm seeing now is this one error that says '
Can't use string " [Just some random zap finding title] .. " as a hash ref while "strict refs" in use at /opt/seccubus/bin/zap2ivil line 153
'
I was wondering if anyone else had some thoughts on what could be wrong with that.

So after digging around I've been able to find out what was wrong. I was using beta versions of some of the scanners and rules and one of the scan rules was dumping some strange stuff into the XML report. All I did was disable it and now seccubus is working like a charm using the python API instead of the .jar file. I didn't have to really get my hands dirty with the code like I though I would, just a few changes and the API can be used instead :ok_hand:


The ensureUpgrades procedure might be broken

Hi, I am running a validator based on the latest client code which is 0.8.24-5cbc418a-x86_64-linux-gnu right now. However, I keep getting "xxx is not running the latest client cod" message.
I thought the reason is the networkId (i.e., Sentry Node Network ID) of these validators is null. Therefore, the ensureUpgrades process will bypass these validators (https://github.com/w3f/1k-validators-be/blob/220bade2643ffbf9458d07a198b4266221b607f3/src/monitor.ts#L65).
The allCandidates and allNodes should be the same lists now since the sentry node is no longer required. (https://github.com/w3f/1k-validators-be/blob/220bade2643ffbf9458d07a198b4266221b607f3/src/db/index.ts#L437).

@lsaether Thank you for your fix.
However, there is a new bug. The process should continue, not return, to check the rest of the nodes.
https://github.com/w3f/1k-validators-be/blob/082d8750b91f205a6671beb751ff06461b52f864/src/monitor.ts#L74
This bug will affect 98% of nodes. There are 3229 nodes in the allNodes list, and the check process stops at the 72nd node because its version is null. I don't know if it's suitable, but I open a PR https://github.com/w3f/1k-validators-be/pull/418 to fix it.


Add code coverage reports

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@3c8c828). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master       #2   +/-   ##
=========================================
  Coverage          ?   88.23%
  Complexity        ?        8
=========================================
  Files             ?        2
  Lines             ?       17
  Branches          ?        0
=========================================
  Hits              ?       15
  Misses            ?        2
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3c8c828...fbc86e8. Read the comment docs.


[Vue warn]: Invalid VNode type: undefined (undefined)

Version
3.1.5
Reproduction link
https://github.com/wenfangdu/el-plus-repro/tree/invalid-vnode-type
Steps to reproduce
Please clone the repro and run.
What is expected?

What is actually happening?

Your reproduction is not minimal, please read and follow https://new-issue.vuejs.org/?repo=vuejs/vue-next#why-repro when reporting a bug. Most bugs should be reproducible with the SFC Playground.


chore(docs): libraries-bom 4.1.0

Released https://search.maven.org/artifact/com.google.cloud/libraries-bom/4.1.0/pom

@elharo  @chingor13 @kolea2 @stephaniewang526 Would you review/merge this update? (I don't have write permission)


Java 4.2: Multidimension array

Description 📜
Add audio, documentation, and video regarding the Multidimension array.

Domain of Contribution 📊

[x] Java

Location of File to be added
The files should be added inside the Multidimension array folder which is inside the Arrays folder present in the  Java main folder.  The documentation file can be in the .md/.py/.ipynb format. Audio should be in .mp3 format and to be submitted through a Google drive link. Video can be in any format and should be submitted through a Google Drive link.
Note:

You are required to comment with \assign CONTENT_TYPE, so that other content types can be created at the same time.
Failing to comment in the above manner will lead to the removal of assignment from the issue even if the bot assigns.
You are required to do a PR within 7 days otherwise you will be unassigned and others will be assigned.
Issues will be assigned on a first come first serve basis.

/assign

/assign Document

@GenMech We cannot assign you this issue as you are already assigned in 4 issues. Please complete them first.

/assign Document

/assign document

@jaigora24 You can clearly see that this issue had already been assigned for documentation content. Why are you spamming all issues to assign documentation to you when they already have been assigned?

If the document has not done yet! Kindly /assign documentation

/assign documentation

\assign Video

@firefistacez You have been assigned video content


changePosition NULL on UnifiedDiffReader.parseUnifiedDiff

I was testing the tool and I picked a random commit from this repo to do so. Let's take https://github.com/java-diff-utils/java-diff-utils/commit/5338313a54c6eed065c3339a71870ba2d6fd0d0f. I accessed the diff of it by adding a .diff at the end of the URL and saved it into diff.txt file. I then used the tool to parse it in the following way:
public static void main(String[] args) throws IOException {
    var stream = new FileInputStream(new File("/path/to/diff.txt"));
    var patch  = UnifiedDiffReader.parseUnifiedDiff(stream);

    patch.getFiles().forEach(f -> {
        System.out.println("File: " + f.getFromFile());
        f.getPatch().getDeltas().forEach(delta -> {

            System.out.println("Source: ");
            System.out.println(delta.getSource().getPosition());
            delta.getSource().getChangePosition().forEach(System.out::print);

            System.out.println("Target: ");
            System.out.println(delta.getTarget().getPosition());
            delta.getTarget().getChangePosition().forEach(System.out::print);
        });
    });
}

The delta getChangePosition is just null. Am I doing something wrong?

Indeed this is missing. It was added into the old parser while the new one was already in progress. I am on it.

@wumpz any eta on when 4.10 will be released? looking forward to it!

I will try it today evening.
-------- Ursprüngliche Nachricht --------Von: Edoardo Lanzini @.> Datum: 28.04.21  09:17  (GMT+01:00) An: java-diff-utils/java-diff-utils @.> Cc: Tobias @.>, Mention @.> Betreff: Re: [java-diff-utils/java-diff-utils] changePosition NULL on
UnifiedDiffReader.parseUnifiedDiff (#117)
@wumpz any eta on when 4.10 will be released? looking forward to it!
—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.

Just released ...
-------- Ursprüngliche Nachricht --------Von: Edoardo Lanzini @.> Datum: 28.04.21  09:17  (GMT+01:00) An: java-diff-utils/java-diff-utils @.> Cc: Tobias @.>, Mention @.> Betreff: Re: [java-diff-utils/java-diff-utils] changePosition NULL on
UnifiedDiffReader.parseUnifiedDiff (#117)
@wumpz any eta on when 4.10 will be released? looking forward to it!
—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.


Failed to minify

With create-react-app default build script, it tries to uglify the dependencies got the following error:
$ react-scripts build
Creating an optimized production build...
Failed to compile.

Failed to minify the code from this file:

 	./node_modules/markdown-it-plantuml/lib/deflate.js:6

Read more here: http://bit.ly/2tRViJ9

Thanks for the feedback! The problem was I was using an es6 feature in deflate.js. Besides, deflate.js was using some undefined variables. I have published a patch to npm. Please, let me know if it works for you =).
Background: that dependency [deflate.js] is a really old implementation and was added as a workaround to keep the library compatible with planuml server, which uses an isomorphic inflate function on its side. I'm looking at deploying my own plantuml server in the long-term, so I did not want to invest time in fixing deflate.js.


Typo on HAPROXY_HTTP_BACKEND_REVPROXY_GLUE template.

Due to a wrong end double quote location in the HAPROXY_HTTP_BACKEND_REVPROXY_GLUE template, it option was generating the haproxy configuration erroneously and making the HAPROXY_{n}_HTTP_BACKEND_REVPROXY_PATH option have unexpected behavior.
I double checked with the HAProxy Documentation
rspirep <search> <replace> [<cond>]

Can one of the admins verify this patch?

@estebanlll thank you for the PR.
LGTM 🚢

Do I have to do something to get it into the next release?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

The bug is still there but seems like nobody used this feature at all. So, should I close the PR or not?

ok to test

@elopezvalecky Applogies for the long delay in getting this reviewed. I'm currently working on getting the backlog cleaned up. I want to merge this in, but it needs to be rebased on master and an additional test fix. I've done this in https://github.com/mesosphere/marathon-lb/pull/623, If you would like to rebase this PR and add in the fix from https://github.com/mesosphere/marathon-lb/pull/623/commits/5cbdb22b294b0123d953efa50a539a19805e2e8d, I'll merge this PR in. Thanks for the contibution!

@jkoelker, I have rebased and added the extra change.
Thanks

Thanks!


is it possible to pick multiple dates?

I want to choose date range in calendar. is this possible?

@sinafbd The range picker feature is the last item on the to-do list. You can check it out on the readme.

@sinafbd The range picker feature is the last item on the to-do list. You can check it out on the readme.

thank you, but when it will release?


Protocol parsing issues on many inserts

Given the schema
CREATE KEYSPACE meow WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};
USE meow;
CREATE TABLE kittehz (id UUID PRIMARY KEY, name VARCHAR, stuff BOOLEAN);

running the latest Cassandra version from Docker on macOS HS
docker run --name some-cassandra -d -p 9042:9042 cassandra

cqlsh:meow> SHOW VERSION;
[cqlsh 5.0.1 | Cassandra 3.11.2 | CQL spec 3.4.4 | Native protocol v4]

executing the code
extern crate cdrs;

use cdrs::client::CDRS;
use cdrs::authenticators::NoneAuthenticator;
use cdrs::transport::TransportTcp;
use cdrs::query::QueryBuilder;
use cdrs::compression::Compression;
use std::error::Error;

fn main() -> Result<(), Box<Error>> {
    let authenticator = NoneAuthenticator;
    let addr = "<IP_ADDRESS>:9042";
    let tcp_transport = TransportTcp::new(addr).unwrap();

    // pass authenticator and transport into CDRS' constructor
    let client = CDRS::new(tcp_transport, authenticator);

    // start session without compression
    let session = client.start(Compression::None)?;

    match session.query(QueryBuilder::new("USE meow").finalize(), false, false) {
        Err(ref err) => panic!("cant use keyspace {:?}", err),
        Ok(_) => {},
    }

    for i in 0..100_000 {
        let query = QueryBuilder::new(r#"
            INSERT INTO kittehz
            (id, name, stuff)
            VALUES
            (uuid(), 'a', false)
        "#).finalize();

        match session.query(query, false, false) {
            Ok(_) => {},
            Err(ref err) => panic!("cant insert {:?}", err),
        }
    }

    Ok(())
}

crashes the library after 15123 inserts, either entering unreachable code
➜  cdrs-bug git:(master) ✗ cargo run --release
   Compiling cdrs-bug v0.1.0 (file:///Users/ath/Desktop/cdrs-bug)
    Finished release [optimized] target(s) in 0.44s
     Running `target/release/cdrs-bug`
thread 'main' panicked at 'internal error: entered unreachable code', /Users/ath/.cargo/registry/src/github.com-1ecc6299db9ec823/cdrs-1.2.3/src/frame/mod.rs:322:18
note: Run with `RUST_BACKTRACE=1` for a backtrace.

or reporting an unexpected Cassandra version

➜  cdrs-bug git:(master) ✗ cargo run --release
    Finished release [optimized] target(s) in 0.02s
     Running `target/release/cdrs-bug`
thread 'main' panicked at 'Unexpected Cassandra version 0, either 4 or 132 is expected', /Users/ath/.cargo/registry/src/github.com-1ecc6299db9ec823/cdrs-1.2.3/src/frame/mod.rs:166:13
note: Run with `RUST_BACKTRACE=1` for a backtrace.

Looks like the parser is getting confused, reading from a wrong offset. I attached a Cargo module / project (what do you even call them?) that reproduces the issue on my machine™.
cdrs-bug.tar.gz

The issue also persists with v2.0.0, with a slightly different error:
➜  cdrs-bug cargo run --release
    Finished release [optimized] target(s) in 0.03s
     Running `target/release/cdrs-bug`
thread 'main' panicked at 'cant insert General("Empty or corrupted frame received")', src/main.rs:38:29
note: Run with `RUST_BACKTRACE=1` for a backtrace.

extern crate cdrs;

use std::collections::HashMap;

use cdrs::authenticators::NoneAuthenticator;
use cdrs::cluster::{Cluster, Session};
use cdrs::query::*;
use cdrs::load_balancing::{RoundRobin};
use cdrs::transport::TransportTcp;

use cdrs::types::prelude::*;
use cdrs::types::from_cdrs::FromCDRSByName;
use cdrs::frame::IntoBytes;
use std::error::Error;

fn main() -> Result<(), Box<Error>> {
    let cluster = Cluster::new(vec!["<IP_ADDRESS>:9042"], NoneAuthenticator {});
    let mut session = cluster
        .connect(RoundRobin::new())
        .expect("No compression connection error");

    match session.query("USE meow") {
        Err(ref err) => panic!("cant use keyspace {:?}", err),
        Ok(_) => {},
    }

    for _ in 0..100_000 {
        match session.query(r#"
            INSERT INTO kittehz
            (id, name, stuff)
            VALUES
            (uuid(), 'a', false)
        "#) {
            Ok(_) => {},
            Err(ref err) => panic!("cant insert {:?}", err),
        }
    }

    Ok(())
}

@athre0z Thank you for reporting this issue. I'm investigating it currently.

@athre0z Unfortunately I was able to reproduce it so far after 5 times running the code.
Last time the fix for very similar issue was to substitute read by read_exact in parser.rs.
Would you be able to check possible solution by replacing following code in parser.rs
    // NOTE: order of reads matters
    let v = try!(cursor.read(&mut version_bytes));
    let f = try!(cursor.read(&mut flag_bytes));
    let s = try!(cursor.read(&mut stream_bytes));
    let o = try!(cursor.read(&mut opcode_bytes));
    let l = try!(cursor.read(&mut length_bytes));

    if v != Version::BYTE_LENGTH || f != Flag::BYTE_LENGTH || s != STREAM_LEN
       || o != Opcode::BYTE_LENGTH || l != LENGTH_LEN {
        return Err(error::Error::from("Empty or corrupted frame received"));
    }

by the possible fix
    try!(cursor.read_exact(&mut version_bytes));
    try!(cursor.read_exact(&mut flag_bytes));
    try!(cursor.read_exact(&mut stream_bytes));
    try!(cursor.read_exact(&mut opcode_bytes));
    try!(cursor.read_exact(&mut length_bytes));

and use your local version of CDRS (cargo.toml):
[dependencies.cdrs]
path = "../path-to-cdrs"

Please let me know if it helped. If so please submit a PR with this solution.
Thank you.

Even though the cursor is based on already filled buffer for me it makes sense to substitute that on as well. So, yes you can go ahead with this one as well


OS X clang can't link against library

My steps:
> cmake .
> make

got libcds.dylib in bin/
> file libcds.dylib
libcds.dylib: Mach-O 64-bit dynamically linked shared library x86_64

trying to compile my project that is using
#include "cds/container/michael_list_hp.h"
#include "cds/container/split_list_map.h"

ld fails with the following error:
Undefined symbols for architecture x86_64:
  "cds::gc::dhp::GuardArray<3ul>::GuardArray()", referenced from:
      cds::gc::DHP::GuardArray<3ul>::GuardArray() in main.o
  "cds::gc::dhp::GuardArray<3ul>::~GuardArray()", referenced from:
      cds::gc::DHP::GuardArray<3ul>::~GuardArray() in main.o
  "cds::gc::dhp::Guard::Guard()", referenced from:
      cds::gc::DHP::Guard::Guard() in main.o
  "cds::gc::dhp::Guard::~Guard()", referenced from:
      cds::gc::DHP::Guard::~Guard() in main.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Can you provide your main.cpp?
Or any usecase with that error.

If it is matter, here is the beginning of main.cpp:
#include <string>
#include <sstream>
#include <unistd.h>
#include "fcgio.h"
#include "pqxx/pqxx"
#include <regex>
#include <algorithm>
#include <ctime>
#include <unordered_map>
#include "cds/opt/hash.h"
#include "cds/container/michael_list_hp.h"
#include "cds/container/split_list_map.h"
namespace cc = cds::container;

using namespace std;

const size_t TIMEOUT = 15*60; //seconds after which the session considered completed

typedef cc::SplitListMap<
	cds::gc::DHP,
	string,
	long,
	cc::split_list::make_traits<      // metafunction to build split-list traits
		cc::split_list::ordered_list<cc::michael_list_tag>,     // tag for underlying ordered list implementation
		cc::opt::hash< std::hash<string> >,        // hash functor
		cc::split_list::ordered_list_traits<    // ordered list traits desired
			cc::michael_list::make_traits<   // metafunction to build lazy list traits
				cc::opt::less< std::less<string> >         // less-based compare functor
		>::type
	>
>::type>  string_int_map;
string_int_map sessions;

Furthermore, there is no difference if I pass the -lcds flag to linker (Other Linker Flags in Xcode project) and put the library nearby, or dont even pass the flag. The linker can not find those symbols.
P.S. Максим, как можно с тобой связаться в более интерактивном режиме?

Pffff, the problem just was that I simply include
#include <cds/gc/hp.h>
instead of
#include <cds/gc/dhp.h>

P.S. Максим, как можно с тобой связаться в более интерактивном режиме?
libcds dot dev at gmail dot com


Upgrade deprecated constructs in sample.py

Duplicate of PR https://github.com/sherjilozair/char-rnn-tensorflow/pull/76.

@julien-c You can close this, it's been merged as part of https://github.com/sherjilozair/char-rnn-tensorflow/pull/81.
https://github.com/sherjilozair/char-rnn-tensorflow/pull/81/files#diff-086ae17ebd1d85217865fc991fdc5fb8L36

Thanks @julien-c and @hugovk, I'm helping @sherjilozair tidy up the repo a bit. If you guys are interested as well please join chat at https://gitter.im/char-rnn-tensorflow/Lobby
First order of business it go through the PR and Issue backlog and clean out the easy stuff before making a pass at bigger items.


Echo Night crashes when selecting "New Game"

Before opening this kind of issue, please ensure:

Are you using a compatible console version? Yes
Are you using CD-R media, and not CD-RW? Yes
Did you copy all the files in the expected format to the memory card? Yes

tonyhax version: 1.4.5
Installation method: Copied files onto a PS1 memory card using DexDrive
Entry point game: CRASH BANDICOOT 2 - CORTEX STRIKES BACK (SCUS-94154)
Console model: SCPH-101
Integrity check: Yes
BIOS version: v4.5 05/25/00 A
Target game: Echo Night (SLUS-00820)
Bug explanation:
The game boots fine, but when I select "New Game" on the title screen, I am taken to a black screen. You can see it happen here. I've also just finished testing it again to be absolutely sure the issue is reproduceable.

Hello. I cannot reproduce your issue on an emulator. Which program did you use to burn the disc?
Redump lists this game as having EDC antipiracy checks, which are data sectors whose checksums are intentionally corrupted, and some burning tools fix automatically.

Hello. I cannot reproduce your issue on an emulator. Which program did you use to burn the disc?
Redump lists this game as having EDC antipiracy checks, which are data sectors whose checksums are intentionally corrupted, and some burning tools fix automatically.

This might be a freepsxboot patch incompatibility (which are not reproducible in emulators).
I really don't trust the EDC listings in redump because I believe they used a tool that has a lot of false positives (since a ton of games have invalid EDC due to the buggy Sony tools not generating the proper EDC/EEC for sectors). The only games I've actually come across with EDC protection are here: https://alex-free.github.io/tonyhax-international/anti-piracy-bypass.html#edc-protected-psx-games

@socram8888 : I used ImgBurn to burn the CD. It's worked perfectly fine for many other PS1 games I've burned. (But I understand that different games may have different antipiracy checks.)
@alex-free: I did notice the message, "Installing patches: * Generic antipiracy * FreePsxBoot" at the bottom of the TonyHax loader thing. Are there any next steps I can do to help with looking into this?
====
Actually, I looked at the list of EDC protected games Alex-free linked to and saw that page also says that ImgBurn won't burn EDC protected games, but CloneCD will. So I will look into using CloneCD, despite Echo Night not being on the official list (yet).
Thanks!

This might be a freepsxboot patch incompatibility (which are not reproducible in emulators).
It should for the original tonyhax - I am unconditionally patching the memory card read process as long as it is a PS1.

This might be a freepsxboot patch incompatibility (which are not reproducible in emulators).

It should for the original tonyhax - I am unconditionally patching the memory card read process as long as it is a PS1.

I think that this issue https://github.com/socram8888/tonyhax/issues/145 did not happen in emulators only on real hardware. I haven't yet had a chance to check this out though.

I don't remember if I tried that issue on an emulator using the 2.0 BIOS he was using. I test on emulators using the 4.4 version my retail PSone has, and there I could certainly reproduce other weird read contention issues. Unsure about this one in particular or the one from #145.

I wasn't able to patch the ISO with EDCRE, even with the -z argument. I might need to track down an ISO with actual different tracks. The one I have might have had its contents consolidated into one bin, I'm thinking.

I wasn't able to patch the ISO with EDCRE, even with the -z argument. I might need to track down an ISO with actual different tracks. The one I have might have had its contents consolidated into one bin, I'm thinking.

I don't think the problem is EDC so EDCRE won't help here

Have you tried using International? It doesn't have the freepsxboot patch so that should work if that truly is the issue

Is the international version of Echo Night just the Japanese version? I'm only seeing two releases here.
https://www.giantbomb.com/echo-night/3030-34532/releases/

Is the international version of Echo Night just the Japanese version? I'm only seeing two releases here.
https://www.giantbomb.com/echo-night/3030-34532/releases/

International is a fork of Tonyhax https://alex-free.github.io/tonyhax-international

Wow, Tonyhax International worked! Thanks for the suggestion!
Here's a link of it in action: https://youtube.com/live/4VvhaF-wcCo
Weird thing I noticed: I did one test before this recording and the CD audio at the beginning of the game was skipping incessantly, but when I did this second test, everything sounded fine. I'm wondering if my PSone was being super picky about how the CD was seated. I place the disc in the tray the same way every time though. ¯_(ツ)_/¯

Wow, Tonyhax International worked! Thanks for the suggestion!
Here's a link of it in action: https://youtube.com/live/4VvhaF-wcCo
Weird thing I noticed: I did one test before this recording and the CD audio at the beginning of the game was skipping incessantly, but when I did this second test, everything sounded fine. I'm wondering if my PSone was being super picky about how the CD was seated. I place the disc in the tray the same way every time though. ¯_(ツ)_/¯

Nice! Btw Ps1s are super picky about the CD-R media and the burner used to burn them. I have a webpage on that: https://alex-free.github.io/psx-cdr
Another thing that can be done is refurbish the cd drive itself: https://alex-free.github.io/unofficial-ps1-cd-drive-service-manual
@socram8888 my theory is this is because of either the fpsxboot patch, or maybe because I do SetMemSize(8) in the bios reinitialize function (I mentioned why I added that before).

Very helpful information! I took a quick peruse and both the media and the drive I'm using aren't so hot. "Fake" Verbatims (the 80 minute ones) and a slim burner drive that came with my Lenovo desktop. (HL-DT-ST DVDRAM GUE0N)
Still, I found it encouraging that it ran fine with no audio skipping on the second try. "Better than nothing", I suppose.
I don't think it's the drive, since I got this PSone in pristine condition in 2021. (But I could be wrong.)

Very helpful information! I took a quick peruse and both the media and the drive I'm using aren't so hot. "Fake" Verbatims (the 80 minute ones) and a slim burner drive that came with my Lenovo desktop. (HL-DT-ST DVDRAM GUE0N)
Still, I found it encouraging that it ran fine with no audio skipping on the second try. "Better than nothing", I suppose.
I don't think it's the drive, since I got this PSone in pristine condition in 2021. (But I could be wrong.)

Hey, if it works for you then great! If you ever encounter issues like skipping FMVs or CDDA though in the future there is somethings that can be done just wanted to let you know. It's also definitely possible that it was just placed a little weird on the spindle on that first go around.

Forgot to mention socram8888 that the setmemsize thing is also not reproducible on emulation (at least with duckstation when I was debugging the gshax issue with ridge racer usa for my fork). My bet is kind of on that not being in your bios reintialize.

@alex-free could be. I am not sure if any of the functions I use do this step as an implicit operation.

I've checked using the debugger and there's no access within my BIOS reinitialization function to the RAM size register, so I've fixed it in 9c127c96b041bb9048dcd05ed93a4a30a443813d.

I've checked using the debugger and there's no access within my BIOS reinitialization function to the RAM size register, so I've fixed it in 9c127c96b041bb9048dcd05ed93a4a30a443813d.

Should be fixed as of latest commit?

Probably, but haven't checked so it's why I left the issue open.


Error in script Palette_inspector_script.gd

There is an error in this script.
Line 3 should read:
var custom_palette_property = preload("res://addons/PaletteTools/scripts/custom_property.gd")
The installation installs that script in ../PaletteTools/script (lowercase s)
on Linux/Ubuntu

I've renamed the Scripts folder to the appropriate casing. You mentioned in the other issue #2  you were having other problems, but you'll have to be specific as it works out of the box on Windows 10 and 11, even when the folder was wrongly cased. I'll look into trying it in a Linux VM later to see if I can find other issues.

I tried it in Ubuntu with Godot 4.2 beta 3 and it works as intended. If you specify more about what problems you're running into, I can try to help.

I tried again installing it on a new project in Godot 4.2 via the AssetLibrary.
The first thing I notice is a message about 3 conflicting files: 2 git files gitignore and gitattributes and the project file project.godot. I have no idea what that is about.
Next I see a boat load of errors in the messages window:
Godot Engine v4.2.dev5.official (c) 2007-present Juan Linietsky, Ariel Manzur & Godot Contributors.
modules/gltf/register_types.cpp:63 - Blend file import is enabled in the project settings, but no Blender path is configured in the editor settings. Blend files will not be imported.
--- Debug adapter server started ---
--- GDScript language server started ---
res://addons/PaletteTools/scripts/palette_inspector_script.gd:3 - Parse Error: Preload file "res://addons/PaletteTools/Scripts/custom_property.gd" does not exist.
res://addons/PaletteTools/scripts/palette_inspector_script.gd:3 - Parse Error: Could not resolve type for variable "custom_palette_property".
modules/gdscript/gdscript.cpp:2656 - Failed to load script "res://addons/PaletteTools/scripts/palette_inspector_script.gd" with error "Parse error". (User)
Attempt to open script 'res://addons/PaletteTools/Scripts/palette_tool.gd' resulted in error 'File not found'.
Failed loading resource: res://addons/PaletteTools/Scripts/palette_tool.gd. Make sure resources have been imported by opening the project in the editor at least once.
Attempt to open script 'res://addons/PaletteTools/Scripts/get_from_url.gd' resulted in error 'File not found'.
Failed loading resource: res://addons/PaletteTools/Scripts/get_from_url.gd. Make sure resources have been imported by opening the project in the editor at least once.
scene/resources/resource_format_text.cpp:284 - res://addons/PaletteTools/Scenes/palette_tools.tscn:153 - Parse Error:
Failed loading resource: res://addons/PaletteTools/Scenes/palette_tools.tscn. Make sure resources have been imported by opening the project in the editor at least once.
res://addons/PaletteTools/palette_tools.gd:14 - Parse Error: Preload file "res://addons/PaletteTools/Scripts/palette_inspector_script.gd" does not exist.
modules/gdscript/gdscript.cpp:2656 - Failed to load script "res://addons/PaletteTools/palette_tools.gd" with error "Parse error".
The plugin shows up under plugins in the project manager but when I try to enable it I get these errors:
Attempt to open script 'res://addons/PaletteTools/Scripts/palette_tool.gd' resulted in error 'File not found'.
Failed loading resource: res://addons/PaletteTools/Scripts/palette_tool.gd. Make sure resources have been imported by opening the project in the editor at least once.
Attempt to open script 'res://addons/PaletteTools/Scripts/get_from_url.gd' resulted in error 'File not found'.
Failed loading resource: res://addons/PaletteTools/Scripts/get_from_url.gd. Make sure resources have been imported by opening the project in the editor at least once.
scene/resources/resource_format_text.cpp:284 - res://addons/PaletteTools/Scenes/palette_tools.tscn:153 - Parse Error:
Failed loading resource: res://addons/PaletteTools/Scenes/palette_tools.tscn. Make sure resources have been imported by opening the project in the editor at least once.
res://addons/PaletteTools/palette_tools.gd:14 - Parse Error: Preload file "res://addons/PaletteTools/Scripts/palette_inspector_script.gd" does not exist.
modules/gdscript/gdscript.cpp:2656 - Failed to load script "res://addons/PaletteTools/palette_tools.gd" with error "Parse error".   core/config/project_settings.cpp:341 - Property not found: editor_plugins/enabled
The addon files are all there in the folders as designated. For example the first error regarding 'palette_tool.gd  File not found'; that file is there and the path is correct.
I have used your plugin on a Windows machine and it works flawlessly. Thank you for that.
I would really like to get this to work on this machine as well.

So I installed again in another new project and repeated the errors as describe prior.
I went back and repopened the other project and the addin begins working.
However I can see no way to download palette files.
I get the custom color picker showing and it shows 'no saved palettes'. But I don't see any menu item relating to downloading palette from lospec.

I would recommend you get it from here at the moment, as I haven't applied for an update in the asset library since making that folder casing change earlier today.
Do the errors appear again after restarting the project? Unfortunately a lot of Godot addons produce a list of errors when first added, but then don't show up on subsequent loads.

Make sure resources have been imported by opening the project in the editor at least once.

This message, which appears next to all those errors, is saying that you need to load it once. It's a very common message when adding a plugin for the first time. Again, it shouldn't appear in future loads. I'm guessing it's just trying to read the files before it's aware it has them.
And referring to your comment:

3 conflicting files: 2 git files gitignore and gitattributes and the project file project.godot

I can restructure the addon so this doesn't happen, but then it no longer works as a full sample project when downloaded. Many addons also have at least the project.godot part so people can just see it's use in a sample project.
This was taken on a 4k display, which is why sizing might appear slightly off:

Just enter your Lospec palette link there and hit the little search icon or enter, and it'll grab the palette.


support/documentation - how to manage multi-level stairs

Suppose I have mutlilevel stairs, i.e. a round stairwell that spans multiple floors.
How can I control that I want to go from floor 1 to 2, then 2 to 3 then 3 to 4, or suppose I'm on 2 and want to go down?
The issue being that the @stair actor is in the same relative location from floor to floor, so the current implementation seems to pick nearly at random.
You don't have to change anything, just document how I can make it so that players can select which direction they want to go. Note that the stairwell is exactly one square wide and high.

Are you saying that you have a set of stairs on a single grid square that goes both up and down?
If so, there isn't any good way to allow choosing at the moment. https://github.com/grandseiken/foundryvtt-multilevel-tokens/issues/23 would make it possible to choose a destination, but for now the typical solution would to be to have the "up" and "down" staircases on different grid squares.

Yes. You can close this one as a dupe of the other.


Script is not resilient to "db crash"

Aug 06 13:07:43 npm-search app/worker.1: [unknown_error: Database encountered an unknown error] {
Aug 06 13:07:43 npm-search app/worker.1:   status: 503,
Aug 06 13:07:43 npm-search app/worker.1:   name: 'unknown_error',
Aug 06 13:07:43 npm-search app/worker.1:   message: 'Database encountered an unknown error',
Aug 06 13:07:43 npm-search app/worker.1:   error: true,
Aug 06 13:07:43 npm-search app/worker.1:   data: '503 Service Unavailable\n' +
Aug 06 13:07:43 npm-search app/worker.1:     'No server is available to handle this request.\n' +
Aug 06 13:07:43 npm-search app/worker.1:     '\n' +
Aug 06 13:07:43 npm-search app/worker.1:     '\n'
Aug 06 13:07:43 npm-search app/worker.1: }
Aug 06 13:07:43 npm-search app/worker.1: error Command failed with exit code 1.

This is something I'm aware of, and don't really have a good solution for in mind. instead of crashing and relying on Heroku to be slow enough, we could back off for e.g. 1 minute & back off more exponentially? I didn't add any code since it seemed to be rare that we didn't wait long enough when crashing

We need to check the error handling with PouchDB, a back off would be good enough I believe as it seems to happen very rarely.

there's a "retry" option, but we should see if it isn't too aggressive https://pouchdb.com/api.html#options-4

Should not happen again


request help: What to do when the downstream protocol is http and the upstream service is tcp

Issue description
Hello:
Our existing services are exposed externally using tcp.
 Now, we plan to migrate the system to k8s deployment,

 Considering that the tcp service is not convenient to expose outside the k8s cluster,

 So we need an http gateway to perform protocol conversion operations, and connected to the back-end tcp service

 We have made a demo based on openrestry + cosocket + ffi to do protocol conversion

 Here, please teachl me how to use apisix to done this work conveniently

Environment

apisix version (cmd: apisix version):
OS: Linux

We have made a demo based on openrestry + cosocket + ffi to do protocol conversion

this way is good. if you need a better performance, we need to hack openresty, we can implement a new cosocket_ffi method.
Of course, for the TCP protocol, multiplexing technology is usually used, which makes the implementation difficult.

We have made a demo based on openrestry + cosocket + ffi to do protocol conversion

this way is good. if you need a better performance, we need to hack openresty, we can implement a new cosocket_ffi method.
Of course, for the TCP protocol, multiplexing technology is usually used, which makes the implementation difficult.

@membphis I think you misunderstood the question. @adugeek want to know how to do this job with Apache APISIX, and he want to make it easier.

@adugeek you can refer to the tcp-logger plugin: https://github.com/apache/incubator-apisix/blob/master/apisix/plugins/tcp-logger.lua, which is also downstream http and upstream tcp.

@adugeek you can refer to the tcp-logger plugin: https://github.com/apache/incubator-apisix/blob/master/apisix/plugins/tcp-logger.lua, which is also downstream http and upstream tcp.

Yes ,I just want to knwon this ! TKS , ^_^.


After Upgrading Angular 9 Getting Karma Error in node_modules , karma - Config.tpl.ts , ERROR in node_modules/karma/config.tpl.ts:13:18 - error TS1109:

Hi Everyone,
I am getting below Karma Config file error after upgrading Angular 9 ,
ERROR in node_modules/karma/config.tpl.ts:13:18 - error TS1109: Expression expected.
frameworks: [%FRAMEWORKS%],
node_modules/karma/config.tpl.ts:13:30 - error TS1109: Expression expected.
frameworks: [%FRAMEWORKS%],
node_modules/karma/config.tpl.ts:17:13 - error TS1109: Expression expected.
files: [%FILES%
node_modules/karma/config.tpl.ts:18:5 - error TS1109: Expression expected.
node_modules/karma/config.tpl.ts:22:15 - error TS1109: Expression expected.
Package.json
"devDependencies": {
"@angular-devkit/build-angular": "~0.900.7",
"@angular/cli": "~9.0.7",
"@angular/compiler-cli": "~9.0.7",
"@angular/language-service": "~9.0.7",
"@types/jasmine": "^3.0.0",
"@types/jasminewd2": "^2.0.0",
"@types/node": "~8.9.4",
"codelyzer": "^5.1.2",
"jasmine-core": "^3.0.0",
"jasmine-spec-reporter": "~4.2.1",
"karma": "^4.0.0",
"karma-chrome-launcher": "^3.0.0",
"karma-cli": "^2.0.0",
"karma-coverage-istanbul-reporter": "~2.0.1",
"karma-html-reporter": "^0.2.7",
"karma-jasmine": "~1.1.2",
"karma-jasmine-html-reporter": "^0.2.2",
"karma-junit-reporter": "^1.2.0",
"node-sass": "^4.13.1",
"protractor": "^5.4.3",
"ts-node": "~7.0.0",
"tslint": "~6.1.0",
"typescript": "^3.6.4"
}

Looks like you compiled a template for a config rather than expanding it and compiling it.


release candidate

I think this version is pretty solid, and hopefully I've fixed all the CI issues...

Aidan gave a verbal OK on the algorithmic improvements; closing so we can move on.  I'd like to see some more testing on exotic platforms... but that's looking like a bit of a project.  Hopefully we get a bug report if something's broken.