Short Term Memory not working

This is the output of the Context appended by Long Term, Short Term and Entities Memory.
["Historical Data:\n- Ensure that the initial 'Thought' statement is removed from the final answer as it may confuse the reader.\n- Maintain consistency in formatting; the final output should not have differing formatting styles in the same section.\n- Add more real-.",
'', # Short Term Element is Empty
'Entities:\n- Learning Rate(Hyperparameter): A crucial hyperparameter in the optimization process that determines the step size at each iteration while moving toward a minimum of the loss function.\n- learning rate(concept): A crucial hyperparameter in AI and machine learning that determines the size of the steps taken towards minimizing the loss function during the training process.\n- Natural Language Processing (NLP)(Field): Involves the interaction between computers and humans through natural language, enabling machines to understand, interpret, and respond to human language.'].
Let me know, If I am doing anything wrong.
In my investigation, Long Term and Entity Memory is giving data to context but short term remain empty.
I have set the memory=True while creating Agent and kicking off crewai object.

Hi, for you is entity memory working fine? means is it consistent for you? For me its not consistent at all. Suppose i am feeding it some list of fruits then it will be able to recognize only one fruit and forgetting every other. Then I tried to do this exercise once again but now it is storing nothing new at all.

I also encountered a situation where the short-term memory consistently returned empty results, while both entities and long-term memory had values. Despite the task types being identical, the short-term memory could not be retrieved. I directly debugged self.app.search() to query the contents of the short-term memory, but it still returned empty

I have learned that Short Term is always empty because each time CreatShortTerm is created with reset=True, it only retains the current task's action. However, during the task execution process, there might be errors in reassigning agents or continuity issues with tasks, which would require using actions to correct errors or optimize the next action. Each creation, however, leads to data loss unless it's a complex task performed correctly, which then becomes meaningless. I believe actions should be created based on Crew rather than agent. (Based on my research and a month of usage, the utilization rate of Short Term is virtually zero, and I hope for some optimization.)

@beeapu


mux: filter for invalid root path pattern

Mux router has trouble with root path matching because of the different semantics of empty path and "/" path. Add the already existing filter into all relevant router functions and add extend test cases to check all combinations.

Thx for contribution :).


Currently I need to retrieve all messages (while deleting them) and then resubmit to the queue.
It will be awesome if there will be a button to handle all of that logic.

I do this now with auto-forwarding.
Given a queue "A" with dead lettered messages. I create a new queue called "B". Then configure queue "A" to forward all dead lettered messages to "B" and then update. All the dead lettered messages should then be in "B" and the DLQ for "A" should be empty. Then configure "B" to forward all messages to "A" and update. All messages will then be back in "A" and be available for processing.

@ahofman I thought about that solution a while back but it doesn't do what I need - what you actually do can be done by setting the number of retries of a message to infinite and ignore the existance of the DLQ.
I do want my messages to go the DLQ under certain conditions, and WHEN I want to resubmit them, I will be able to do it with ease.
@ahofman do you agree or am I missing something?

I don't think we're quite on the same page - I am not suggesting ignoring the DLQ or anything like that. I'll try again but with more detail.
Let's say you have a queue called "A". This queue is configured with Max Delivery Count of 10. There were 100,000  messages in this queue that have been received 10 times, and their delivery count has exceeded the Max Delivery Count, and all the messages have been dead-lettered. So what we have in this scenario is a queue called "A" with 0 active messages and 100,000 messages in the dead letter queue for "A". The objective is to resubmit all 100,000 messages from the DLQ in "A" such that they are active again in "A", with their delivery counts reset, so that they can be processed again.
To achieve this, create a new queue called "B". Then configure queue "A" to forward all dead-lettered messages to "B", like so (don't forget to click the Update button):

After refreshing the queues, you will see 100,000 active messages in "B" and 0 dead lettered messages in A. The active messages in "B" will have a Delivery Count of 1.
Now configure queue "B" to auto-forward all messages to "A", like so (don't forget Update):

We'll now see 100,000 active messages in "A" with a delivery count of 1, ready to be processed - mission accomplished :) (unless of course I've missed something!)
Note in this scenario we've set up a loop of automatic retries of dead lettered messages in "A" - hence we should turn off the auto-forwarding once we've completed the retry operation.

Yes that make more sense when you do it ad-hoc and then disable it.
The thing is that it takes time and if I have 10 queues with stuck dead messages? It will take a long time to acomplish.
The manual process I do currently takes a bit less time but if I had to deal with 100k dead messages I will defiantly use your method.
I still think it will make much more sense to have a single button to perfrom the operation in the UI - it will take 5 seconds to handle 10s of queues!

Hi @regevbr Service Bus Explorer is an open-source tool and is supported by the community on a best-effort basis. Unfortunately, in the last couple of years I've been quite busy and I've a limited time to dedicate to the tool. I think that your idea is good, but at the same time your issue is not so common. In other words, it would be great if you could implement the feature you suggest and submit a PR. Otherwise, I can look into it, but not now, because I don't have time in this period, sorry about that :(

I completely understand! Last I touched c# was a few years back and I'm a node.js guy now but if i will find the time I will try to give it a shot

thanks @regevbr ;)


pl.readRecords(rows, {schema}) crashes when a schema is specified

Have you tried latest version of polars?
[yes]
What version of polars are you using?
0.7.3
What operating system are you using polars on?
Mac M1 on Ventura 13.2.1
What node version are you using
node v19.2.0
Describe your bug.
.readRecords() method crashes when called with a Schema.
What are the steps to reproduce the behavior?
const pl = require('nodejs-polars');

const rows = [
  { num: 1, date: "foo", string: "foo1" },
  { num: 1, date: "foo" },
];

const schema = {
  num: pl.Int32,
  date: pl.Utf8,
  string: pl.Utf8,
};

const df = pl.readRecords(rows, { schema });

What is the actual behavior?
thread '<unnamed>' panicked at 'called `Option::unwrap()` on a `None` value', src/conversion.rs:613:37
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5

What is the expected behavior?
Since the provided code is a copy-paste from the tests, here is the entire test set:
test("from row objects, with schema", () => {
    const rows = [
      { num: 1, date: "foo", string: "foo1" },
      { num: 1, date: "foo" },
    ];

    const expected = [
      { num: 1, date: rows[0].date.toString(), string: "foo1" },
      { num: 1, date: rows[1].date.toString(), string: null },
    ];

    const schema = {
      num: pl.Int32,
      date: pl.Utf8,
      string: pl.Utf8,
    };
    const df = pl.readRecords(rows, { schema });
    expect(df.toRecords()).toEqual(expected);
    expect(df.schema).toEqual(schema);
  });

Using  "nodejs-polars": "^0.8.0" and JS, this works:
const schema = {
  num: pl.Int32,
  date: pl.Utf8,
  string: pl.Utf8,
};

const df = pl.readRecords(rows, schema );

Or this:
const schema = {
  num: pl.DataType.Int32,
  date: pl.DataType.Utf8,
  string: pl.DataType.Utf8,
};

const df = pl.readRecords(rows, { schema });

@Jonarod Can you please try this code in 0.8.1? If it works, please close the ticket. Thx

This code works fine in 0.8.2, please re-open is this still an issue. Thx
const schema = {
  num: pl.DataType.Int32,
  date: pl.DataType.Utf8,
  string: pl.DataType.Utf8,
};

const df = pl.readRecords(rows, { schema });


[FEATURE REQUEST] Provide input (of all types) state control outside of Form scope

This function solves the problem (这个功能解决的问题)
Currently it's not possible to set the invalid / valid state of an input (Red border on the field) at will. It is controlled by the form-field and form components. The issue here is that if we do not wish to use the built-in solution for validation / form management, we have no solution to set the state of a field.
Expected API (期望的 API)
Adding a prop to each input field (Input, Select, DatePicker, TimePicker, ...) that provides optionnal control on the state of the field, like this :
<n-input state="error" />

This prop could take any of the main states already implemented in buttons for example (error, warning, info, success, ...)

We will finally add it but it would take some time.


Typescript: Improve gen code to install npx playwright install

For typescript project, when generate code, we miss following step npx playwright install so sometimes generated-code is not ready to run test (see image)

TODO:  Fix here https://github.com/rockmelonqa/rockmelonqa-ide/blob/c6cc01700d996e226be41477834b53af1d1097da/electron/worker/actions/buildCode.ts#L43

I haven't been able to reproduce this, so I think it's been fixed


Creating bare repository skips storage namespace validation

Some bad things can happen as a result, e.g.

Creating a repo with no storage access
Creating multiple repositories under the same namespace

hi, please assign me the issue if it's available

hi, please assign me the issue if it's available

Done!

I see that the mentioned validations (write access check and checking for existing repo at the same storage namespace) occur right now only for non-bare non-readonly repos.
I guess that these checks should be added only for non-readonly bare repos?

@idanovo any reason why we didn't add namespace validation on read only repos?

@idanovo any reason why we didn't add namespace validation on read only repos?

@N-o-Z
I don't think read-only repositories are relevant to this issue; they are not the same as bare repositories.
This issue has existed since before the announcement of the read-only repositories feature.
Am I missing something?

@idanovo any reason why we didn't add namespace validation on read-only repos?

@N-o-Z Yes- since this is a read-only repository, there is no harm in case the storage namespace we are using is already used by another repository or if we don't have write permissions for this namespace.
We can have for example one "regular" repository for this namespace and another read-only repo(s) for this namespace.
Maybe we should document this one better as it took me some time to remember we had done it.

@N-o-Z https://github.com/treeverse/lakeFS/pull/8396

I would like to understand why this is the right thing to do.  Creating a bare repo is intentionally a plumbing operation, and I am afraid of limiting it, especially at this late stage.
For instance, if we make this change then restoring a repository becomes harder: I need to worry about whether dummy already exists in the repository.
This needs a few minutes of face-to-face discussion; fortunately we have some coming up this week!


Links to files in release are broken

Hi, I've tried downloading the files in the release (https://github.com/yerfor/GeneFace/releases/tag/v1.0.0), but the links for lrs3 dataset and May.mp4 seem to be broken.

Hi, this link will be available in several days, along with a new version of 3DMM extractor.
Actually, I'm trying to turn to a PyTorch-based 3DMM extractor and uploading a new LRS3.zip based on this extractor. Since the google drive capacity is limited, I have to empty the space temporarily.

About two days.

Hi, the new codebase and dataset files are available. Therefore I will close this issue.


DOH UAT - Data missing for 6 NYS Counties

On NY COVID Data tab, there is no data for Warren, Washington, Wayne, Westchester, Wyoming, and Yates Counties.
App originally crashed when these counties were selected. A fix was put in to prevent app from crashing if there is no data, but the data for these counties is still missing.

This is fixed on the dev API (see example screenshot for Wayne), so will be fixed when that update reaches the production API.

@mikegwhit Can you re-assign this back to Kevin when the prod api data is correct? Thanks

This issue should have been fixed CC @kevin-sheldon
We may still need to push to prod.

I now see county data in the test app. Let me know if there is anything that the NF team needs to do before we close this.

@kevin-sheldon No, you can close this, thanks


Deletion of File when Renaming File

Issue Type: Bug

Create an empty directory using Window File Explorer.
Launch VS Code on the created directory.
In VS Code Explorer create a text file named "foobar.txt".
In VS Code add some content to the text file.
In VS Code save the text file.
In Windows File Explorer open the text file with Windows Notepad to verify it's really there and has the content you added.
In VS Code Explorer rename the text file to "FooBar.txt".
Poof your text file has been deleted rather than being renamed.

Error: ENOENT: no such file or directory, rename '...\foobar.txt' -> '...\FooBar.txt'
This should be a high priority fix due to data loss.
VS Code version: Code 1.67.2 (c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5, 2022-05-17T18:15:52.058Z)
OS version: Windows_NT x64 10.0.19044
Restricted Mode: No

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz (16 x 2904)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: enabledopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okskia_renderer: enabled_onvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
31.74GB (20.01GB free)

Process Argv
--crash-reporter-id 73cd0d11-ee50-4449-98fd-87da60c458d0

Screen Reader
no

VM
0%

Extensions (33)

Extension
Author (truncated)
Version

vscode-sqlite
ale
0.14.0

es6-string-css
bas
0.1.0

markdown-mermaid
bie
1.14.0

insert-unicode
bru
0.13.3

vscode-standardjs-snippets
cap
0.9.0

excel-to-markdown-table
csh
1.3.0

vscode-markdownlint
Dav
0.47.0

vscode-eslint
dba
2.2.2

xml
Dot
2.5.1

vscode-pandoc
Dou
0.0.8

gc-excelviewer
Gra
4.2.54

vscode-guid
hea
1.4.20

rest-client
hum
0.24.6

plantuml
jeb
2.17.2

vscode-insertdatestring
jsy
2.3.1

marp-vscode
mar
1.5.2

rainbow-csv
mec
2.4.0

vscode-language-babel
mgm
0.0.36

vscode-html-format
moh
0.0.4

remote-containers
ms-
0.234.0

remote-wsl
ms-
0.66.3

cpptools
ms-
1.9.8

vetur
oct
0.35.0

vscode-print
pdc
0.9.14

vscode-powerquery
Pow
0.1.32

lit-plugin
run
1.2.4

markdown-preview-enhanced
shd
0.6.3

vscode-standard
sta
2.1.0

code-spell-checker
str
2.2.0

pdf
tom
1.2.0

literally-html
web
0.1.3

markdown-pdf
yza
1.4.4

cursorCharCode
zei
0.2.0

A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythonvspyl392:30443607
pythontb:30283811
pythonptprofiler:30281270
vsdfh931cf:30280410
vshan820:30294714
vstes263:30335439
pythondataviewer:30285071
vscod805:30301674
pythonvspyt200:30340761
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593cf:30376535
vsc1dst:30438360
pythonvs932:30410667
wslgetstarted:30449410
pythonvsnew555:30457759
vscscmwlcmt:30465135
cppdebug:30492333
vsclangdf:30486550

I cannot reproduce.
/gifPlease

Possibly related? #149287

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!


made namespace name match project-id

AFAIK, we decided that namespaces should be specific qualified names not tied to any customer set information. Is this still current?


SKE Enablement logic fails on a fresh project for the first time

Hi,
I am deploying to a "fresh" STACKIT project and the first time I ran my terraform script I got the following error:
╷
│ Error: Error creating cluster
│
│   with stackit_ske_cluster.ske,
│   on main.tf line 202, in resource "stackit_ske_cluster" "ske":
│  202: resource "stackit_ske_cluster" "ske" {
│
│ Wait for SKE enablement: 404 Not Found, status code 404, Body: {"code":"NotFound",
│ "message":"project id \"6d519696-52b3-45ac-9cc1-3d95831005aa\": not found", "details":""}
│
╵

Running it again the enablement logic seemed to work and it created the cluster successfully.
Expected behaviour:
It should run through without errors the first time.

Hey @PeterStolz, thank you for reaching out!
Sorry for the inconvenience, we are aware of this issue and plan to address it this week.
Regards,
Diogo

Awesome. Thanks for the quick response


Fixes bundled output path

Without this, specifying a bundled output path will fail. Particularly important for projects that have multiple outputs (you don't want your builds overwriting each other).

Good catch! Thanks for the PR.


Update key validation to support PHP 5.4. Fix #75.

Thanks for the PR @lfbittencourt - note that we don't support PHP 5.4, only the versions supported here:
http://php.net/supported-versions.php
Regardless, this is a completely forwards-compatible change, so it's been merged in!

To be honest, I'm running an outdated Vagrant box and thought "yeah, it won't hurt, right?" :stuck_out_tongue:
Thanks for merging in!


Flag -n is not useful on pull and push

kubect tag push and pull accept a -n flag that is not needed as we derive the namespace from the image one is pulling/pushin.

Solved.


Add rng accessor so a specific seed/rng can be used

Hello! This PR adds a couple of methods to allow a user to specify a Random object as the random number generator, which makes the gem usable if you need some sort of deterministic randomness (which I do!)

I ended up not using this gem, and probably won't have time to work on these changes, so you can probably close this PR. Thanks!


Attach existing branch to ticket, attach multiple branches to ticket

Many times during development, a ticket might be multi-purposed and might need multiple branches. Also, sometimes needed branches already exist, but are not yet associated with a particular ticket. It would be incredibly helpful to have that as an option. Doing what's recommended here:
https://stackoverflow.com/questions/25789652/jira-assign-an-existing-git-branch-to-an-issue
can introduce errors, and could result in a huge loss of work if done incorrectly.

Hi @showduhtung. You can add multiple branches to one Jira issue so long as the branch contains the issue key. If there are existing branches that don't include the issue key, they would need to be renamed to be identified by Jira and connected to the correct ticket.

Let me know if I've understood your issue so I can close this out/follow up.

As it has been almost 2 months with no response, closing this issue.


feat(rust): show interactive menu to select one or more TCP inlent when user dosen't specifies

Make ockam tcp-inlet delete (no args) interactive by asking the user to choose from a list of tcp-inlet aliases to delete using tuify
After this PR gets merged and if no one takes (#6465, #6464, #6463 and #6462) I'll apply the same changes in one or multiple othr PRs to close them

Closes: build-trust/ockam/issues/6466

Current behavior
Shows an error message when the user didn't specify an alias

Proposed changes

Make the alias argument optional
Check if the user specifies an alias or not
if so, then do the old behavior (check, confirm, delete)
if not, and the terminal is interactive, and the user didn't pass any of --no-output or --quiet flags

then collect all exists TCP inlets aliases in the (selected or default) node
Shows them in a multi selection TUI menu
confirm and delete

Checks

[x] All commits in this Pull Request are signed and Verified by Github.
[x] All commits in this Pull Request follow the Ockam commit message convention.
[x] There are no Merge commits in this Pull Request. Ockam repo maintains a linear commit history. We merge Pull Requests by rebasing them onto the develop branch. Rebasing to the latest develop branch and force pushing to your Pull Request branch is okay.
[x] I have read and accept the Ockam Community Code of Conduct.
[x] I have read and accepted the Ockam Contributor License Agreement by adding my Git/Github details in a row at the end of the CONTRIBUTORS.csv file in a separate pull request to the build-trust/ockam repository. done via: build-trust/ockam-contributors/pull/266

Hi @0x61nas , thanks for submitting this PR! We recently merged the first PR related to making some commands interactive at https://github.com/build-trust/ockam/pull/6480
That PR can serve as a guide on how to use some common functionalities that are now available as part of the Terminal struct. Make sure to rebase latest develop. And let me know if you have any questions!

@adrianbenavides Hi, is there any plan to review this anytime soon(in this October)?. Hope you have a nice day.

@0x61nas while we still have to review this change. I'm adding the HACKTOBERFEST-ACCEPTED label so the contribution is counted by hacktoberfest bots before the end of October. Thank you for taking time to contribute 🎉
Please not @adrianbenavides's feedback above.

It's looking great @0x61nas ! We have two similar PR's related to other show commands where we are using a different structure to handle the multiple use cases. My only request for now would be to adapt your PR to that structure (check #6547 and #6665). Let me know if you have any questions!

fine, i'll follow it

Ho @0x61nas , your PR will be merged through https://github.com/build-trust/ockam/pull/6886 Thanks again for your help!


Jar with source code not generated as part of build process

Having a jar with source code can be useful for debugging. The following plugin should be added to the pom.xml file to generate the source code jar:

org.apache.maven.plugins
maven-source-plugin

attach-sources

jar

Can you please put the pom file content that we should add?

No activity on this for a while. Closing.


Wrapped Type generates unexpect column name in select sql

When using Wrapped Type, quill append value to column name in select sql. I guess this is the same behavior as mentioned here https://github.com/getquill/quill/issues/114#issuecomment-174273740 and would be fix in https://github.com/getquill/quill/pull/369?
I'm raising the issue here anyway, so we don't miss it.
Version: 0.6.0
Module: quill-sql
Expected behavior
Given
  case class Id(value: Long) extends AnyVal with io.getquill.sources.WrappedValue[Long]
  case class Account(
    id : Id,
    name     : String
  )

I expect quill to generate
SELECT x.id, x.name FROM Account x
Actual behavior
quill appends value to id column
SELECT x.idvalue, x.name FROM Account x
Steps to reproduce the behavior
object QuillExample2 extends App{
  case class Id(value: Long) extends AnyVal with io.getquill.sources.WrappedValue[Long]
  case class Account(
    id : Id,
    name     : String
  )

  def main() = {
    val db = source(new SqlMirrorSourceConfig("db"))
    db.run(query[Account])
  }
}

Workaround
Add
implicit val encodeId= mappedEncoding[Long, Id](Id(_))
@getquill/maintainers

I guess this is the same behavior as mentioned here https://github.com/getquill/quill/issues/114#issuecomment-174273740 and would be fix in https://github.com/getquill/quill/pull/369?
I'm raising the issue here anyway, so we don't miss it.
And do you see any issue with workaround above? I've tried it with simple CRUD and seems to be ok

@komsit37 Thank you for the report. It should be fixed by https://github.com/getquill/quill/pull/364. Could you verify if it works with the 0.6.1-SNAPSHOT version? I'm closing the issue, but please don't hesitate to reopen it if necessary.

yup 0.6.1-SNAPSHOT works fine. thanks!


Support MinIO S3 with TLS enabled (self-signed certificate)

I'm using s3fs to access a self-hosted MinIO S3 server via below code:
fs = s3fs.S3FileSystem(
        key="MyKey",
        secret="MySecretPassword",
        client_kwargs={"endpoint_url": "http://<IP_ADDRESS>:9000"},
    )

This works as intended when I do not enable TLS. I have then set up my MinIO server with a self-signed certificate to enable TLS. When doing so, I've attempted the below process:
os.environ["SSL_CERT_FILE"] = "C:\\Users\\marti\\.minio\\certs\\public.crt"

fs = s3fs.S3FileSystem(
        key="MyKey",
        secret="MySecretPassword",
        client_kwargs={"endpoint_url": "https://<IP_ADDRESS>:9000"},
        use_ssl=True
    )

In the MinIO S3 SDK, TLS is handled by parsing the certificate as per the extra line above. But this does not seem to have any effect on my result - I get the below error when I try to run my script with the MinIO server and TLS enabled:
Traceback (most recent call last):
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connection.py", line 344, in connect
    ssl_context=context)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\util\ssl_.py", line 357, in ssl_wrap_socket
    return context.wrap_socket(sock)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\ssl.py", line 412, in wrap_socket
    session=session
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\ssl.py", line 853, in _create
    self.do_handshake()
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1056)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\httpsession.py", line 263, in send
    chunked=self._chunked(request.headers),
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\util\retry.py", line 343, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\packages\six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connection.py", line 344, in connect
    ssl_context=context)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\util\ssl_.py", line 357, in ssl_wrap_socket
    return context.wrap_socket(sock)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\ssl.py", line 412, in wrap_socket
    session=session
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\ssl.py", line 853, in _create
    self.do_handshake()
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1056)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\marti\Downloads\packages\canedge_browser\examples\list_log_files.py", line 59, in <module>
    example_list_files_for_devices_between_dates_s3()
  File "C:\Users\marti\Downloads\packages\canedge_browser\examples\list_log_files.py", line 47, in example_list_files_for_devices_between_dates_s3
    fs, devices, start_date=start_date, stop_date=stop_date
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\canedge_browser\listing.py", line 103, in get_log_files
    session_list = fs.ls(device, detail=True)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\s3fs\core.py", line 529, in ls
    files = self._ls(path, refresh=refresh)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\s3fs\core.py", line 426, in _ls
    return self._lsdir(path, refresh)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\s3fs\core.py", line 334, in _lsdir
    for i in it:
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\paginate.py", line 255, in __iter__
    response = self._make_request(current_kwargs)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\paginate.py", line 332, in _make_request
    return self._method(**current_kwargs)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\client.py", line 622, in _make_api_call
    operation_model, request_dict, request_context)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\client.py", line 641, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\endpoint.py", line 137, in _send_request
    success_response, exception):
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\endpoint.py", line 256, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\retryhandler.py", line 251, in __call__
    caught_exception)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\retryhandler.py", line 317, in __call__
    caught_exception)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\endpoint.py", line 269, in _send
    return self.http_session.send(request)
  File "C:\Users\marti\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\httpsession.py", line 281, in send
    raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://<IP_ADDRESS>:9000/ce2-target?list-type=2&prefix=23AD1AEA%2F&delimiter=%2F&encoding-type=url [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1056)

I guess the issue is related to the underlying use of boto3 in s3fs - and that boto3 might not have a way to parse a self-signed certificate. If anybody else has encountered issues with this use case, I would be interested in your thoughts on how to solve it.
Martin

Does it fail already when connecting, or is it a specific operation that causes the exception?
It seems to me that you are supposed to specify not just use_ssl=True, but verify="/path/to/cert" in the client kwargs (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.client )

Hi martindurant, thanks for the quick reply!
It seems that using your tip as below solved the issue:
    fs = s3fs.S3FileSystem(
        key="CANedgeTestServerAccessKey",
        secret="MySecretPassword",
        client_kwargs={
            "endpoint_url": "https://<IP_ADDRESS>:9000",
            "verify": "C:\\Users\\marti\\.minio\\certs\\public.crt",
        },
        use_ssl=True,
    )

I'll double check that this works as intended - thanks again!

Excellent

Your input solved it for me Martin, thanks for your fast response and great support!


Support for reMarkable Paper Pro

I know the rMPP is brand new and most people (including myself) haven't received theirs yet, but I'm kinda surprised nobody has asked this yet.
Are you planning to update goMarkableStream to support the rMPP?
If so, you may want to add something to the repo's README.md file saying this.

Hello;
I would if I had a Rmpp, but to honnest, I don't plan buying one.
So I will unlikely update the tool to support the rmpp.
Maybe if a friend buy one, and I can put my hands on it I could evaluate the work to do to make it compatible.
Meanwhile, if someone works on it,  he can contact me to gain admin rights on the repo.

Hi @owulveryck, my RMPP is arriving next week (Sep 30). I'd like to help you access the tablet -- remotely through SSH, or by giving you a filesystem/memory dump -- to update goMarkableStream. We have overlapping work, since I will need to get RMPP screenshots working in RCU, and both of our projects would benefit by our working together. Please send me an email if you'd like to set something up.

I will get mine in a few days and I was literally planning to build something like this (in golang as well), will be happy to work on the rmPP support if necessary.

Hi @davisremmel @owulveryck had you have a chance to sync on that and see if this would work ?

Addendum, copying binary 0.8.5 via SCP, then running, I get:
root@imx8mm-ferrari:~# ./goMarkableStream -unsafe -sh: ./goMarkableStream: cannot execute binary file: Exec format error

Addendum, copying binary 0.8.5 via SCP, then running, I get: root@imx8mm-ferrari:~# ./goMarkableStream -unsafe -sh: ./goMarkableStream: cannot execute binary file: Exec format error

I got the same issue (downloading the binary and the scp it to RMPP). I think it may be solved recompiling the application for the specific SoC.
However the application is designed for the form factor of RM2 (resolution is hard coded in the first 2 rows here: https://github.com/owulveryck/goMarkableStream/blob/main/client/main.js). I think may be solved at build time extracting this in a configuration and then creating a build specific for each platform, or at runtime, if possible, reading the model type and modifying configuration according to this.
I don't know Go Language but the server side main function  is easily readable (https://github.com/owulveryck/goMarkableStream/blob/main/http.go): there is a middleware that streams the screen of the Remarkable via Websockets. This code seems to be reusable on the RMPP but I suppose that you need to check if the low libraries are present and usable also in the RMPP.
I don't have an idea of what are the steps to rebuild the app, it should be useful a tutorial or similar documentation.

The application is in Go, therefore you can cross compile it on your laptop (one of the reason I choose Go).
Actually the application was designed for RM2. My guess is that it can be tweaked to work on RMPP, but I have no RMPP, so I can test it.
If you want to play, I hope that the articles on my blog can help you get a picture first and then expand it to the application.
Good luck to you.

Hi, how crosscompile for Pro? I'm a Java guy, I don't know how to get binary platform and how to compile.

The application is in Go, therefore you can cross compile it on your laptop (one of the reason I choose Go). Actually the application was designed for RM2. My guess is that it can be tweaked to work on RMPP, but I have no RMPP, so I can test it.
If you want to play, I hope that the articles on my blog can help you get a picture first and then expand it to the application.
Good luck to you.

Thank you for your reply. I waited to answer because I needed to create a Linux VM on my Macbook.
I read all you articles (well done a very good job) I don't understand everything cause I am new to Go language. Following your instructions I downloaded this repo (and Go compiler as well). After changing screen resolution in main.js (2160x1620 according to specifications of RMPP) I succedeed recompiling the application without errors, but after installing and running it I found the same message "Exec format error".
The command I used is the one you write in the Readme:
GOOS=linux GOARCH=arm GOARM=7 CGO_ENABLED=0 go build -v -trimpath -ldflags="-s -w" .
Then I changed the architecture to arm64
GOOS=linux GOARCH=arm64 GOARM=7 CGO_ENABLED=0 go build -v -trimpath -ldflags="-s -w" .
And this time it starts (so remember Paper Pro is a 64 bit architecture), but of course if an app starts it does not automatically mean that it works. :)
Let's forget for a moment the certificate issue (not a real issue, I use a local IP address), going further it still does not work. The terminal output gives this error:
"open ./testdata/full_memory_region.raw: no such file or directory"
So i "touched" this file. Now the terminal output does not give an error, but still showing nothing in the browser window.
I observed that:

opening a new document and writing with the stylus has the effect that the log terminal shows an "EOF" message.
debugging Javascript I can confirm that Javascript recognize that the tablet is in portrait or landscape.
there are no Javascript errors.

So I am stalled at this point. I don't understand where is the error, maybe it si related to the full_memory_region.raw (I still didn't analyzed where is located in your code and why it does not create automatically this file) or maybe there is an issue in the /stream endpoint (how can I debug it? Which tool did you used?) or there is something else that not works.
Can you help me? Maybe with four eyes we can solve this issue. :)

Nice job do far.
Maybe we could arrange a Zoom/Meet call for 30 minutes. What is your Time zone? I am on holiday this week, maybe next week. WDYT?

Sorry for the late answer but I had to work much last week.
Thank you but a call It is not necessary (and my spoken English is very poor) :)
But still I appreciate your help, I explain my last analysis.
First, after analysing code I understood that the app was trying to access the file "./testdata/full_memory_region.raw" because the source fb_arm.go is not valid for architecture arm64 and the compiler defaulted to fb.go (which have a dummy method that I suppose you used this for testing on your PC). Because I don't know what is the right filename (not fb_arm64.go) I simply copied the code of the function "GetFileAndPointer" in fb.go (a dirty solution I know :) ) and now it works.
No, it's not really working. :)
But I finally found the real problem: there is no /dev/fb0 framebuffer reference in /proc/{id}/map in Paper Pro. In other words the command:
cat /proc/{id}/maps | grep /dev/fb0
gives nothing. After some looking around I tried /dev/dri/card0, I think it's probably wrong.
I tried via command line to generate an image raw with the command (I am using you guide, the size is 2160x1620):
  dd if=/proc/441/mem of=image.raw count=3499200 bs=1 skip=281472945901568
but for now I don't find a way to convert the raw to an image.
Anyway I substituted the address in the code (brutally), recompiled and now it shows an "image". And it changes, too, when I write with the stylus. In the attachments you will find the screenshot and the image raw (you must rename it).

I hope you can help me, also with some suggestions.
Thank you in advance.

@danielealbano /sys/class/graphics is an empty folder. But also finding the stride value, I don't have idea about how to convert the raw image removing the stride.

Hi.
i am also interested to get this running on a rmPP.
@cristiannobili  what memory address did you actually use?
i am not able to read any of the /dev/dri/card0 mapped areas of the process.
every region is resulting in dd giving me immediately an I/O error.
However i was able to convert the image.raw  into something meaningful. If i shorten the width by 6 pixels. please see the python script below
import sys
import cv2
import numpy as np
if sys.stdin.isatty():
    print ("i need data to convert")
    sys.exit(0)

buf=sys.stdin.buffer.read()
array=np.frombuffer(buf,dtype=np.uint8)
buf_len=len(buf)
skip=0
skip_line=0
width=2154
lines=(buf_len-skip)/(width+skip_line)
height=int(lines)
out=[]
array=np.empty((width,height),dtype=np.uint8)
pos=skip
for y in range(height):
    for x in range(width):
        pos+=1
        pix=buf[pos]
        out.append(pix)
    pos+=skip_line
out=bytearray(out)
array=np.frombuffer(out,dtype=np.uint8)
array=np.reshape(array,(width,height))
cv2.imwrite("out.png",array)

When running it
cat image.raw | python3 raw_to_png.py

i get the following output

So the image is there. Great job.  So I converted the value:

echo $((0xffffa1bd8000))
=><PHONE_NUMBER>94208

Then I copied the memory dump to raw file. In this case the proc id is 463 so:

dd if=/proc/463/mem of=image.raw count=3499200 bs=1 skip=281473395294208
and you get the raw file.
The next steps to do are:
a) modify pointer.go in order to obtain the right address of memory area.
Something like this:
file, err := os.OpenFile("/proc/"+pid+"/maps", os.O_RDONLY, os.ModeDevice)
	if err != nil {
		log.Fatal("cannot open file: ", err)
	}
	defer file.Close()
	scanner := bufio.NewScanner(file)
	scanner.Split(bufio.ScanWords)
	scanAddr := false
	var addr int64

	for scanner.Scan() {
		if scanAddr {
			hex := strings.Split(scanner.Text(), "-")[0]
                        addr, err = strconv.ParseInt("0x"+hex, 0, 64)
                       addr += 26357760
                       break
		}
		if scanner.Text() == `/dev/dri/card0` {
			scanAddr = true
		}

	}

the "magic number" is obtained subtracting the last address minus the first (the memory size of framebuffer is constant). Tested it, it's the right value.
b) convert the image inside the application using go language.
Here I need some help.

Many thanks.. it worked for me as well.
it is really interesting, that the end of the last address  area is actually the start of image :-)
i think we have 3 challenges left:
Challenge #1:
the go issue you mentioned (I never worked in go so far, hopefully i find some time drilling me in)
Challenge #2:
we are missing the color information.
below you see an image that i grabbed from my rmpro. I wrote all the colors (sorry for my handwriting :-)) in their respective colors.
In the memory area i have the following value counts:

value
count

0 ("black")
23778

2
1

53
1

144
1

255 ("white")
3474315

i would have expected somehow different values to indicate some color infos..
Challenge #3:
above the "grey" i wrote "black". (see photo) This is missing in the memory-dump...
i assume the "skip" address is to far off...

Agree.
But I think that first we have to get goMarkable working even if only in bw.
If you change the code using the snippet above now it sends a  scrambled image to the webpage and it changes when you write, so it is a great step forward, so we are close the get something working. :)
I isolated the component that does the job:
// Write encodes the data using run-length encoding (RLE) and writes the results to the subwriter.
//
// The data parameter is expected to be in the format []uint4, but is passed as []byte.
// The result is packed before being written to the subwriter. The packing scheme
// combines the count and value into a single uint8, with the count ranging from 0 to 15.
//
// Implements: io.Writer
func (rlewriter *RLE) Write(data []byte) (int, error) {
	length := len(data)
	if length == 0 {
		return 0, nil
	}
	buf := bufferPool.Get().([]uint8)
	defer bufferPool.Put(buf)

	current := data[0]
	count := uint8(0)

	for i := 0; i < remarkable.ScreenHeight*remarkable.ScreenWidth*2; i += 2 {
		datum := data[i]
		if count < 254 && datum == current {
			count++
		} else {
			buf = append(buf, count)
			buf = append(buf, current)
			current = datum
			count = 1
		}
	}
	/*
		for i := 0; i < remarkable.ScreenWidth*remarkable.ScreenHeight; i++ {
			datum := data[i*2]
			if count < 254 && datum == current {
				count++
			} else {
				buf = append(buf, count)
				buf = append(buf, current)
				current = datum
				count = 1
			}
		}
	*/
	buf = append(buf, count)
	buf = append(buf, current)

	return rlewriter.sub.Write(buf)
}

I don't studied yet how to resolve this ( I work on this project only at late evening) but playing with parameters I suspect that there is another issue: when I change page the webpage does not refresh, or better, when I create a new page it does not refresh. Maybe it's a gesture not recognized.
I observed also another phenomenon. When you approach the pen without touching the screen, the app receives some event and start refreshing the screen. I think this behavior could be specific of Paper Pro, because here there is an active stylus with a battery (maybe in future Remarkable could activate new functions, like laser pointer or similar).

Wohhh congratulations to all of you.
I am sorry, I have very little time to work on this in this period..
I  say again that I am willing to offer help via a zoom/meet call of one hour or so. (@cristiannobili no worries, with your english and my italian, we may find a way to understand each others).
Regarding the last comment, I have implemented a mechanism of idleness to spare bandwidth and battery. The stream stops after 2 seconds or so, and start again when there is an activity with the pen. This must be the reason.

The RLE is a compression mechanism, maybe for a first attempt, you can try to bypass it.
To do so, you can change this line: https://github.com/owulveryck/goMarkableStream/blob/087878dfbd8d7f432c6010f5d260f99fae2fa916/internal/stream/handler.go#L107 and use w instead of the rlewriter (there are both implementation of io.Writer that should do the trick).
Then, you must change the web component as well to write the code directly into the image without the RLE decoding process (you need to modify this: https://github.com/owulveryck/goMarkableStream/blob/087878dfbd8d7f432c6010f5d260f99fae2fa916/client/worker_stream_processing.js#L53)

I had one hour, so I played with image.raw.
This go code generates a picture:
package main

import (
 "encoding/binary"
 "image"
 "image/png"
 "log"
 "os"
)

func main() {
 palette := make(map[uint8]int64)
 spectre := make(map[uint8]int64)
 testdata := "../testdata/image.raw"
 // testdata := "../testdata/multi.raw"
 stats, _ := os.Stat(testdata)
 f, err := os.Open(testdata)
 if err != nil {
        log.Fatal(err)
 }
 defer f.Close()
 backend := make([]uint8, stats.Size())
 log.Printf("backend is %v bytes", len(backend))
 n, err := f.ReadAt(backend, 0)
 if err != nil {
        log.Fatal(err)
 }
 log.Printf("read %v bytes from the file", n)
 picture := backend[:2160*1620]
 boundaries := image.Rectangle{
        Min: image.Point{
                X: 0,
                Y: 0,
        },
        Max: image.Point{
                Y: 2154,
                X: 1624,
        },
 }
 img := image.NewGray(boundaries)

 for i := 0; i < len(img.Pix); i++ {
        if i*2+2 > len(picture) {
                break // Avoid out-of-bounds access
        }
        img.Pix[i] = uint8(binary.LittleEndian.Uint16(picture[i*2 : i*2+2]))
 }
 for i := 0; i < len(picture); i += 2 {
        spectre[picture[i]]++
 }
 for _, v := range img.Pix {
        palette[v]++
 }
 log.Println(spectre)
 log.Println(palette)

 png.Encode(os.Stdout, img)
}

BUT:

why is the resolution 2154x1624
there are two pictures...

I will have a closer look if I have time.
Thank you all for your work

Can one of you provide an image.raw with the colors ? Maybe @To-Si ?
Thank you

sorry for my radio-silence..
with the help from @cristiannobili  i think i found the memory area:
# map of the xochitl process..
 cat /proc/432/maps | grep dri
ffff9688a000-ffff96a37000 rw-s 00000000 00:05 199                        /dev/dri/card0
..... some lines deleted ....
ffff9c2ac000-ffff9c459000 rw-s 00000000 00:05 199                        /dev/dri/card0

# the end address of the last line is the start of the area @cristiannobili found...
cat /proc/432/maps | grep ffff9c459000
ffff9c2ac000-ffff9c459000 rw-s 00000000 00:05 199                        /dev/dri/card0
ffff9c459000-ffff9d9f0000 rw-p 00000000 00:00 0

# lets dump the whole area
dd if=/proc/432/mem of=/tmp/colorful.raw skip=281473303547904 count=22638592 bs=1024 iflag=skip_bytes,count_bytes

please find attached the complete 22 MB raw dump of the colorful image attached which i posted earlier.
I have shared an PNG of the the first 3499200 bytes of that memory area. However the black writing was missing. I believe this is somehow a color-mask used internally to show all non-black colors.
Unfortunately i have not the time to scan through the complete 22 MB to see if we somehow find also the color-coded image.
I am deeply under water at the moment, sorry that i cannot do more analysis right now.
colorful.raw.gz

at the position 8605705 into the colorful.raw from above i have found really interesting stuff...
This is clearly the top part of image (black, gray, and the top of blue). however the pixel-format and the scaling is broken, but we are getting closer

ok, now we are really close... i think the only missing piece is if the skipping of 8605705  is always the same number..
import sys
import cv2
import numpy as np
import pprint as pp
if sys.stdin.isatty():
    print ("i need data to convert")
    sys.exit(0)

buf=sys.stdin.buffer.read()
skip=0
width=2154
height=1624
skip=8605705
array=np.empty((width,height,3),dtype=np.uint8)
pos=skip
for x in range(width):
  for y in range(height):
        array[x,y,0]=buf[pos+3]
        array[x,y,1]=buf[pos]
        array[x,y,2]=buf[pos+1]
        pos+=4
cv2.imwrite("out.png",array)

``
when i run the colorful.raw through the above python code i get the results in this png
![colorful](https://github.com/user-attachments/assets/de18cdaa-b70e-455f-b715-987f4642ce02)

I think, i have decoded the memory footprint. The
The data is chunked. Each chunk start with a 16 byte header.
At position 8 we find the chunk size (inclusive header) as little endian.
After the 16 byte header you find the payload.
Then the next chunk starts.
For unknown reasons the chunk size is 2 bytes to big, I just substracted these 2 bytes from the offset calculation.
With the following python script we should be able to calculate the correct offset of the screenbuffer and grab the screen
import sys
import cv2
import numpy as np
if sys.stdin.isatty():
    print ("i need data to convert")
    sys.exit(0)

buf=sys.stdin.buffer.read()
offset=0
width=2154
height=1624
length=2
# we seek for the chunk that has room for 2154*1624 4-byte pixels
while length<width*height*4:
    # for whatever reason we need to substract 2 bytes from the length
    offset+=length-2
    # at pos 8-16 it contains the chunk (header + payload length)
    header=buf[offset+8:offset+16]
    length=int.from_bytes(header,"little")

# we found the screen
# dump it to png
offset+=16 # skip header
array=np.empty((width,height,3),dtype=np.uint8)
for x in range(width):
    for y in range(height):
        array[x,y,0]=buf[offset+2]
        array[x,y,1]=buf[offset]
        array[x,y,2]=buf[offset+1]
        offset+=4
cv2.imwrite("screen.png",array)

I think, i have decoded the memory footprint. The The data is chunked. Each chunk start with a 16 byte header. At position 8 we find the chunk size (inclusive header) as little endian. After the 16 byte header you find the payload. Then the next chunk starts. For unknown reasons the chunk size is 2 bytes to big, I just substracted these 2 bytes from the offset calculation.

The length is most likely is a int16 or uint16 which doesn't include itself, what I find curious though is that the actual data in memory is in the BGR format instead of RGB556 or similar, which is fairly common for these kind of displays.
Although it might not be there visible, can you check the content of /sys/class/drm/card0/{output} (or similar)? The {output} is the output name used internally, not sure what's going to be called, perhaps card0-DSI-1 or perhaps something else instead of DSI as the screen most likely will be using a different kind of connector.
Also, it's very interesting that the device is not visibile in /dev/ considering that it's opened, perhaps it's possible to find the fd in /proc/{process_id_with_mmapped_device}/fd with ls -lha, it should be a symlink.
To "protect" the access, the boot sequence might (a) espose the device within the initrd, let X / wayland start, and then switch to root or (b) let X / wayland start and then remove the physical inode, which wouldn't impact the running applications as they have the fd already opened ... or perhaps they are using a patched kernel to make it invisible unless you are a special user.

Here is the corresponding Go code to extract a picture from the colorful.raw. This is a strict adaptation of the code to @To-Si
package main

import (
	"bytes"
	"encoding/binary"
	"fmt"
	"image"
	"image/color"
	"image/png"
	"io"
	"log"
	"os"
)

func main() {
	// Read input data
	buf, err := io.ReadAll(os.Stdin)
	if err != nil {
		fmt.Println("Error reading input:", err)
		os.Exit(1)
	}

	width, height := 2154, 1624
	offset := 0
	length := 2

	// Find the chunk with enough room for pixels
	// Recherche d'un morceau de données suffisamment grand pour contenir les pixels
	for length < width*height*4 {
		// Avancer l'offset pour atteindre le prochain morceau
		offset += length - 2

		// Extraire l'en-tête de 8 octets à partir de l'offset
		header := buf[offset+8 : offset+16]

		// Utiliser binary.Read pour lire un uint32 à partir de l'en-tête
		var length32 uint32
		err := binary.Read(bytes.NewReader(header[:4]), binary.LittleEndian, &length32)
		if err != nil {
			log.Fatal(err) // Gérer l'erreur si la lecture échoue
		}
		length = int(length32)
	}

	// Skip header
	offset += 16

	// Create image
	img := image.NewRGBA(image.Rect(0, 0, width, height))

	// Extract and mirror pixel data horizontally
	for x := 0; x < width; x++ {
		for y := 0; y < height; y++ {
			// Calculate the mirrored x-coordinate
			mirroredX := width - x - 1

			// Get the pixel index from the original image
			idx := offset + (x*height+y)*4

			// Set the pixel color at the mirrored x-coordinate
			img.Set(mirroredX, y, color.RGBA{
				R: buf[idx+2], // Red channel (from original image)
				G: buf[idx],   // Green channel (from original image)
				B: buf[idx+1], // Blue channel (from original image)
				A: 255,        // Alpha channel (fully opaque)
			})
		}
	}

	if err := png.Encode(os.Stdout, img); err != nil {
		fmt.Println("Error encoding PNG:", err)
		os.Exit(1)
	}
}

i have posted a bug above...
once we are  getting the correct offset address, the RGBA mapping becomes straight forward...
        R=buf[offset]
        G=buf[offset+1]
        B=buf[offset+2]
        A=buf[offset+3] # this would be my guess.. did not check
        offset+=4

@owulveryck why did you mirror the X axis? and isn't it sufficient to do the mirroring calculation in the outer X-loop prior the Y-loop?

import os
import sys
import subprocess

try:
    PID = subprocess.check_output(["pgrep", "xochitl"]).decode().strip()
except subprocess.CalledProcessError:
    print("Error: Process 'xochitl' not found.")
    sys.exit(1)

print(f"Process: {PID}")

maps_file = f"/proc/{PID}/maps"

memory_range = None
with open(maps_file, "r") as f:
    lines = f.readlines()

    for i in range(len(lines) - 1, -1, -1):
        if "/dev/dri/card0" in lines[i]:
            if i + 1 < len(lines):
                memory_range = lines[i + 1].split()[0]
            break

start, end = memory_range.split("-")
start = int(start, 16)
end = int(end, 16)
count = end - start
length = 2
offset = 0
width = 2154
height = 1624

with open(f"/proc/{PID}/mem", "rb") as mem_file:
    while length < width * height * 4:
        offset += length - 2
        mem_file.seek(start + offset + 8)
        header = mem_file.read(8)
        length = int.from_bytes(header, "little")

skip = start + offset

print(f"Count: {count}")
print(f"Skip: {skip}")
print(f"Extract: dd if=/proc/{PID}/mem count={count} bs=1024 iflag=skip_bytes,count_bytes skip={skip}")
print(f"Remote: ssh<EMAIL_ADDRESS>dd if=/proc/{PID}/mem count={count} bs=1024 iflag=skip_bytes,count_bytes skip={skip} | ffmpeg -f rawvideo -pixel_format rgba -video_size 1624x2154 -i pipe:0 -frames:v 1 screenshot.png")

Script to automatically get the rgba contents and offsets needed. Run on the device. :)

I still can't seem to get the HTTP server to serve any valid images.
These are the files I modified:

I modified pointer.go to grab the correct memory offset.
I also tried removing the RLE compression by editing handler.go and worker_stream_processing.js, but I'm not sure if I did it correctly.
I modified constants.go with the new dimensions.
Do we need to make any modifications to raw.go? It seems like the raw data is being sent directly without being encoded into a PNG.

If you still have the time I think we can arrange a meeting of some sort though I don't have much experience in go @owulveryck

Impressive work by everyone involved! I've combined these efforts into a branch to experiment further and this is what I got. It's possible to get a raw screenshot directly from the device running the app (no streaming yet, unfortunately). Here's a quick guide:
Build and copy the binary to the device:
make build-remarkable-paper-pro
scp goMarkableStream root@{remarkable.local}:~/

Run it on the device:
RK_DEV_MODE=true ./goMarkableStream

# RK_DEV_MODE is required for /raw to work

Run the client tool on your computer to make a screenshot
go run tools/client/client.go -ip {ip} -output screenshot.png

Hi Alexander,
great work.
i have crawled through your code (total dry run at the moment, i am lacking a go-setup)
Some thoughts:

I think the multiplication _ width x size x 2  _ is only done to ensure worst-case memory demand for RLE encoding
At the same time i think that for the paper-pro the RLE algorithm needs to reworked signifcantly. the current implementation will only produce useful output when the image contains large white areas. All other content will be trashed by likely different value for either color channel. I assume the Alpha Channel is always 0, so this will really kill the usefulness of the  current RLE.

How to proceed? As a quick circumvention I would replace the RLE encoder and decoder by a dummy that is simply returning the raw image.
Going beyond this, a downsampling of 4-byte RGBA to 2-byte RGB might be a quick win.
Any thoughts?

Why not mjpeg? With a quality ratio of 80/85, the single images would be fairly small and wouldn't really lose quality and long can be used for screenshot, to preserve more details
Both can easily generated in go and potentially accelerated via NEON (although sending data from the neon simd registers to memory is slower than avx and unless the computation makes it worth it's better to avoid it).
A alternative might be to generate a video stream going through webrtc (which can also be easily used via go) but I imagine this would be a major change.

I give a short answer for mjpeg and co.
When I developed the tool for RM2 I wanted its footprint on the device to be the lowest possible (for battery and so on).
This is the reason why I did not encode it in jpeg nor I used a good compression algorithm.
That being said, I want to thank and congratulate all of you for the work so far.

Did you get the chance the measure the power consumption for mjpeg?
I understand the assumption that it would consume more because the extra complexity of the code but it doesn't translate automatically to actual additional power consumption or the extra power consumption might be irrelevant.
Additionally, for jpeg it's possible to use libjpeg-turbo, which supports NEON, and it's several times faster than libjpeg.
Using such external library with CGO wouldn't have any performance impact when encoding an entire frame as the extra nanosecond needed by the CGO jumps are noting compared to the ms needed for the encoding.

I updated my branch, fixed the streaming, added a note that RLE does now work for Paper Pro, and for now added optional zstd compression in addition to gzip, so the streaming is a bit faster.
If you build it and then run this on the device, it should work:
ZSTD_COMPRESSION=true RLE_COMPRESSION=false ./goMarkableStream

It's all a bit hacky, and I'm not sure that Remarkable 2 isn't broken since I can't test it.

I will try to test it on RM2
Many thanks for your work.
Is there someone willing to handle the maintenance of the code for RMPP ?

Congratulations to all of you.
OSS rocks !

I will try to test it on RM2

Thanks @owulveryck! I opened a PR, feel free to change whatever is needed.
Again, not everything is working as expected on Paper Pro (I added at least a few things to the PR), so I'd really appreciate it if someone else could also test it 😄

Hi...
i hacked together a small RLE adaptation for the rmppro screens.
This could be done either solo (to keep it close to the goMarkableStream) or as a pre-processor to zlib.  (dunno how much the gain would be)
the idea:

we downsample 32-bit RGBA to 15-bit RGB with each channel 5 bit.
we store this 15-bit RGB in a 16-bit value.. and encode it like this C RRRRR GGGGG BBBBB
now there is a trick: we use the unused bit (C above) to indicate if the the value is a count or not

examples
count not set
we read the follow values (1 * 2 bytes)
0 11111 11100 11110
This means that we have no repeat, so we have 1 pixel with R=31, G=28, B=30.
count set
we read the follow values (2 * 2 bytes)
1 00000 00000 00100 0 11111 00000 11111
This means that we have a count, which is 4, then we read the color-value R=31, G=0, B=31.
We write times the pixel 4 times into out.
When writing to output we simply explode the pixel from 15 bit to 32 (or 24) bit as needed.
The colorful.raw is compressed by from 13992384 to 426408 (or compression ratio of 97%)
python PoC of the RLE encoder and decoder
# execute with
cat colorful.raw | python3 extract_screen_rle.py

import sys
#import cv2
import numpy as np

class RLE:
    def __init__(self):
        return

    def rgb555(self):
        pos=self.inpos
        buf=self.inbuf
        r=buf[pos]
        g=buf[pos+1]
        b=buf[pos+2]
        r5=r>>3
        g5=g>>3
        b5=b>>3
        rgb=r5<<10
        rgb+=g5<<5
        rgb+=b5<<0
        return rgb

    def write_rle(self,count,rgb):
        if count==1:
            self.outbuf[self.outpos]=rgb
            self.outpos+=1
            return
        tmp=count+32768
        self.outbuf[self.outpos]=tmp
        self.outbuf[self.outpos+1]=rgb
        self.outpos+=2

    def write(self):
        # same color? just increase counter
        if self.rgb==self.old_rgb:
            self.count+=1
            return
        # different color.. we need to write RLE
        tmp=self.count
        while tmp >=1 :
            # we write max of 32767 colors
            tmp2=min(tmp,32767)
            self.write_rle(tmp2,self.old_rgb)
            tmp-=32767
        self.count=1

    def encode(self,inbuf,inbuf_s):
        self.inbuf=inbuf
        self.inpos=0
        self.outbuf=np.empty(inbuf_s,dtype=np.uint16)
        self.outpos=0
        self.old_rgb=self.rgb555()
        self.count=0
        while self.inpos<inbuf_s*4:
            self.rgb=self.rgb555()
            self.inpos+=4
            self.write()
            #print(f"{self.inpos=} {self.outpos=} {self.old_rgb=} {self.rgb=} {self.count=}")
            self.old_rgb=self.rgb
        self.write()
        ratio=self.outpos/self.inpos
        print(f"compressed from {self.inpos} to {self.outpos} bytes. Ratio={ratio}")
        return (self.outbuf,self.outpos)

    def decode(self,inbuf,inbuf_s,outbuf_s):
        outbuf=np.empty(outbuf_s*4,dtype=np.uint8)
        inpos=0
        outpos=0
        while inpos<inbuf_s:
            inp=inbuf[inpos]
            #print(inp)
            if inp<32767:
                count=1
                inpos+=1
            else:
                count=inp-32767
                inp=inbuf[inpos+1]
                inpos+=2
            #print(f"decode {inp=} {count=} {inpos=} {outpos=}")
            for i in range(count):
                r=(inp>>10)&31
                g=(inp>>5)&31
                b=(inp)&31
                a=0
                r=r<<3
                g=g<<3
                b=b<<3
                outbuf[outpos]=r
                outbuf[outpos+1]=g
                outbuf[outpos+2]=b
                outbuf[outpos+3]=a
                outpos+=4
        print(f"decoding done {inpos=} {outpos=}")
        return outbuf

if sys.stdin.isatty():
    print ("i need data to convert")
    sys.exit(0)

buf=sys.stdin.buffer.read()
offset=0
width=2154
height=1624
length=2
# we seek for the chunk that has room for 2154*1624 4-byte pixels
while length<width*height*4:
    # for whatever reason we need to substract 2 bytes from the length
    offset+=length-2
    # at pos 8-16 it contains the chunk (header + payload length)
    header=buf[offset+8:offset+16]
    length=int.from_bytes(header,"little")

# we found the screen
# dump it to png
offset+=16 # skip header
rle=RLE()
comp,count=rle.encode(buf,width*height)
decomp=rle.decode(comp,count,width*height)
comp2,count2=rle.encode(buf,width*height)
decomp2=rle.decode(comp2,count2,width*height)
print(f"{count=} {count2=} {width*height} {width*height*4} {len(decomp)}")
for i in range(len(decomp2)):
    if (decomp[i]!=decomp2[i]):
        print(f"nonmatch at {i} {decomp[i]} != {decomp2[i]}")

I merged all the code into the main branch and released v0.19-beta with attached the two binaries (for RMPP and RM2).
You can should be able to test it on both devices.
Than you again for all your amazing work. Enjoy the end of the year.

Works pretty well on my Paper Pro in some initial testing, with one minor caveat: attempting to run the process with an ePub or PDF file open produces failed to get memory range: no mapping found for /dev/dri/card0.  Running the process with a notes file or on one of the other screens (settings, file listings, etc) works fine.  Opening a PDF or ePub file with the process already running works as well--the only issue appears to be starting the process with a file already open.
I believe this is caused by the presence of the xochitl_pdf_renderer process when a PDF or ePub file.  The findXochitlPID function gets the PID of that process instead of the core xochitl process, and starts looking in the wrong place for the framebuffer.
I've created a PR, #124, with a fix that should only return the correct xochitl process.  I haven't tested it on a ReMarkable 2 yet--only on a Paper Pro.

Thank you;
It has been merged and v0.19-beta2 has been released

@AngleOSaxon had you considered sub-processing out to pgrep rather than walking a number of files in /proc ?

@AngleOSaxon had you considered sub-processing out to pgrep rather than walking a number of files in /proc ?

No; I don't do a lot of Go or *nix work, and was aiming for the shortest path from current state to fixed.

to each their own, but I'd suggest os/exec might be more expected.
The command we'd like is pgrep -f ^xochitl$
func getProcessByName(processName string) string {
	// Run the `pgrep` command
	cmd := exec.Command("pgrep", "-f", fmt.Sprintf("^%s$", processName))
	var out bytes.Buffer
	cmd.Stdout = &out
	if err := cmd.Run(); err != nil {
		// If no process is found, pgrep exits with 1, which is not an error in this context
		if exitError, ok := err.(*exec.ExitError); ok && exitError.ExitCode() == 1 {
			return "" // No processes found
		}
		return "" // Other errors
	}

	// Parse the output
	lines := strings.Split(strings.TrimSpace(out.String()), "\n")
	for _, line := range lines {
		pid, err := strconv.Atoi(line)
		if err != nil {
			continue // Skip invalid lines
		}
		return strconv.Itoa(pid)
	}
	return ""
}

Not that it matters but it's about twice as fast when checked via time -v ./findcmd2 vs traversing via virtual proc file system, mostly due to switching overhead. To be sure it's a micro-optimization. There might even be a more golang centric way without exec or directory traversal.

@Lewiscowles1986: Thanks so much for the suggestion! 😊
I’m aiming to make the software fully self-sufficient, which is why I prefer a pure Go implementation. If pgrep were ever removed from the base utilities on the reMarkable, the software might stop working, and I’d like to avoid that.
That said, this code only runs once at startup, so while I always appreciate optimizations, this one probably wouldn’t make much of a difference to the user experience.
Thanks again for sharing your thoughts! 🚀

is there a tick list of to-do list we can attach to issue to see progress / opportunity for contribution, or is this complete?
I see:

[X] adjust to screen size of rmpp
[X] detect PID of xochitl
[X] Read libdrm framebuffer from rmpp
[ ] Build instructions for folks uncomfortable with compiling


Common methods for error & alert endpoint

@itsankit-google @rmstar Please review.

Similar methods have been merged through another PRs.


Fix race condition in Porch

Original issue URL: https://github.com/kptdev/kpt/issues/3661
Original issue user: https://github.com/mortent
Original issue created at: 2022-11-16T05:16:55Z
Original issue last updated at: 2022-11-16T05:16:55Z
Original issue body: Porch is using the go-git library from multiple threads (background poller and the requests) and it seems like the library is not thread safe. Example of how to reproduce this problem in https://github.com/GoogleContainerTools/kpt/pull/3602
Original issue comments:

Probably already fixed by https://github.com/kptdev/kpt/pull/3900

Yes, this seems to be fixed, the code mentioned in https://github.com/kptdev/kpt/pull/3602 has been refactored.


⚠ Samsung has degraded performance

In 4b67d6a, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2387 ms

Resolved: Samsung performance has improved in 0ba6e94.


/server uses a vulnerable version of node-fetch and should be patched >= 2.6.7

I see recent changes in /example were made to resolve node-fetch to >= 2.6.7 (CVE-2022-0235)
Can we apply the same for /packages/server so that it includes<EMAIL_ADDRESS>simplewebauthn/packages/server/package.json
"dependencies": {
    .
    .
    .
    "node-fetch": "^2.6.0",
  },

Since PR #68 these libraries specifically use carets for versioning so that projects using the library remain free to incorporate security fixes for things like node-fetch without my needing to cut a new release. It's also why I don't include package-lock.json files in server or browser, unlike /example which I'm free to update via npm audit fix just like any other application that consumes this library.
Are you unable to run npm audit fix and update your project to<EMAIL_ADDRESS>because of simplewebauthn/packages/server/package.json?

I'm going to close this out for lack of feedback. Please feel free to reopen if you believe it's still an issue.


If you do not have access to some files in the workspace, Ionide will crash

Describe the bug
If you do not have access to some files in the workspace, Ionide will crash
Steps to reproduce

Create a folder in the project folder
Revoke access to that folder
Open Ionide

System.AggregateException: One or more errors occurred. (Access to the path '/home/runner/project/testing/bar.txt' is denied.)
 ---> System.UnauthorizedAccessException: Access to the path '/home/runner/project/testing/bar.txt' is denied.
 ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at System.IO.FileStatus.ThrowOnCacheInitializationError(ReadOnlySpan`1 path)
   at System.IO.FileSystemInfo.Create(String fullPath, String fileName, Boolean asDirectory, FileStatus& fileStatus)
   at System.IO.Enumeration.FileSystemEnumerableFactory.<>c.<FileInfos>b__5_0(FileSystemEntry& entry)
   at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext()
   at Microsoft.FSharp.Collections.Internal.IEnumerator.choose@240.System.Collections.IEnumerator.MoveNext() in D:\a\_work\1\s\src\FSharp.Core\seq.fs:line 251
   at Microsoft.FSharp.Collections.SeqModule.ToArray[T](IEnumerable`1 source) in D:\a\_work\1\s\src\FSharp.Core\seq.fs:line 1027
   at Microsoft.FSharp.Collections.ArrayModule.Parallel.Collect@2047-1.Invoke(Int32 i) in D:\a\_work\1\s\src\FSharp.Core\array.fs:line 2047
   at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`1.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)

Expected behaviour
It should ignore files it cannot access
Machine info

OS: Linux
.NET SDK version: 7.0.401
Ionide version: 7.13.0

could you give some more of the FSAC logs (in the "F#" output pane in VSCode) here? We can see the stack trace but none of the stack frames point to which method or function in FSAC is throwing here.


{SQL} fixes Azure/azure-cli#23885 - commands.py

fixes Azure/azure-cli#23885
az sql db import/export commands can take so long to run that it will time out the CLI or DevOps pipeline leveraging AzureCLI.
SO this PR addresses this issue by adding the --no-wait parameter to these commands.
Related command

Description

Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change
[Component Name 2] az command b: Add some customer-facing feature

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] The PR title and description has followed the guideline in Submitting Pull Requests.

[x] I adhere to the Command Guidelines.

[x] I adhere to the Error Handling Guidelines.

SQL

@evelyn-ys Thanks for your comments. Your suggestions have been incorporated. Please review the PR once you get a chance.


Feature/database-activator

Hi! This PR is a working implementation of the database activator, however it needs quite a few improvements. I don't have enough time to improve the database implementation, however I need this part in some projects. I'd therefore like this PR to be merged already so other can also improve the database implementation.

Hi,
Thanks for your contribution!
I can't merge this yet because of:

failing tests
missing tests for the new feature.

You can however already use this in your project, you don't have to have your code merged.
https://stackoverflow.com/questions/13498519/how-to-require-a-fork-with-composer

great feature, looking forward to use it

I agree with @nWidart the LaravelModulesServiceProvider can't have that kind of if statement. What happens if I'm not using a database in my application? It's always checking the Schema to see if it exists.
I'd also correctly name the database config to modules or something on them lines.

was looking for something related, if this PR is added to the package it would be very good


Upgrade to match lexical 0.16.0

It's been a while since I updated this library, but I should get to it soon.

Completed with 644a8a96d339698be2179d7adfaa917d0261c195 and released in ed6417b6e6c7e70a91e183131ab34b6fc77517d5!


Update README to use ocaml-mdx binary

Following https://github.com/realworldocaml/mdx/pull/110, use of the mdx binary now outputs an error message:
Warning: 'mdx' is deprecated and will one day be removed.
    Use 'ocaml-mdx' instead

This fixes the README to use the non-deprecated binary ocaml-mdx instead.

Thanks!


UI-less $audio.record

The current UI of $audio.record is too rigid and can't do anything other than the features dictated by the library we're using.
If we can turn $audio.record into a UI-less action, it would be easy to create all kinds of different UI on top of it.

Since this is coming up, i'd like to add to the discussion the features i've been looking forward to in an audio library. not all of them are applicable to $audio.record, but i mention them because they require underlying flexibility that is a benefit anywhere audio is processed.

accessible buffer on both input and output

e.g. audio for playback is decoded to the buffer and played from it. this enables gapless playback. play/pause affects playback of the buffer, not queuing to it.
e.g. 2: recorded audio is added to the buffer before regularly being flushed to storage. the application can choose to flush at any time, and could even perform non-blocking operations on the data in the buffer for EQ, effects, visualizations, or sending raw audio to a server.

personally, i would like to use this to record audio in to separate gapless files of 7s each for life-logging, but i know this is a severe edge-case

treat each audio stream as separate, like it's an instance of a class. this enables sending data to any output, sending multiples at once (even by merging before sending if the output device can only understand one stream), and other features
don't limit the bitrate or sampling frequency

this allows for future formats
if the on-device DAC can't handle higher frequencies they can just be downsampled anyway
also allows for external DACs that can handle it


Add What's new for .NET 8

Contributes to dotnet/docs#39412

Internal previews

📄 File
🔗 Preview link

dotnet-desktop-guide/net/wpf/migration/index.md
How to upgrade a WPF desktop app to .NET 7

dotnet-desktop-guide/net/wpf/whats-new/net80.md
dotnet-desktop-guide/net/wpf/whats-new/net80

Thanks @gewarren


Configure CI/CD for release notes changelog

Goal
We would like to have the automated changelog for release notes (as for QFI).

See #212.
Considerations:

Development tag 0.0.0-ci.${COMMITHASH}.
Do NOT release on NPM right now because we need to sync w/ Vee first.


bump to Xcode 11.0

Have not touched the simulators yet, as they are not released yet? Can update as soon as there's a official release.

Yeah. Waiting for that final GM to migrate 'mojave-flutter' to Xcode 11 🙌

Do you think there is a chance to get a Xcode 11 image going using the GM2 seed at least? That's the recommended version at this tiny window. Cirrus does have a flutter image using beta5.. was hoping we can update that one quickly?

@fkorotkov hi - I think we are ready to bump the version?

Yeah. mojave-xcode-11 base image was updated to the GM over the weekend. Mergin it now and will initiate a process of building and rolling the new mojave-xcode-11-flutter image. Will update mojave-flutter image alias to the new version as well.


Remove Google Helpouts from tutoring services

It was shutdown on April 20, 2015 https://support.google.com/helpouts/answer/6168100?rd=1

Good catch, thanks!


to improve the stock market data scraper code

Issue Description:
The current implementation of the stock market data scraper code lacks error handling and validation for user inputs. This can lead to potential crashes or incorrect results when users provide invalid input. Additionally, the code could benefit from modularization and improved formatting of stock information and data.
Proposed Changes:
Implement error handling and validation for user inputs:

Add functions to validate the date format and ensure it matches "YYYY-MM-DD" format.
Validate the period input to ensure it is a positive integer.
Handle exceptions when retrieving stock market data and display appropriate error messages.
Modularize the code:

Create separate functions for getting user input, retrieving stock data, printing stock information, and displaying stock data.
Encapsulate related code within functions for better organization and maintainability.
3. Improve formatting of stock information and data:
Use the shutil.get_terminal_size().columns method to dynamically adjust the centering of printed output based on the terminal size.
Format and display stock information in a more visually appealing manner.
Enhance the display of stock market data by formatting it appropriately.

@avinashkranjan
I am a GSSOC contributor
I want to work on this issue
Please assign it to me

Go Ahead @anupammaurya6767

@avinashkranjan pls take a look


In addition to "score" and "life" add "level"

This might need some user testing but the idea is to make it easier to swap between tilemaps in a game. You often want to create several for different stages.

To clarify, this doesn't need to be in the info category; it could be a scene API or something.


Adding wso2is:v6.1.0 clustering support

Purpose

Adding support for a clustered deployment of WSO2 IS
Fixing issues faced when using clustering with WSO2 IS v6.1.x with the operator
Added support for different ways of applying configs
[x] As inline toml configs
[x] As a ConfigMap applied to the cluster separately
[ ] As custom fields in the CRD yaml
[x] Adding watchers to watch for changes to ConfigMap to trigger reconciliation
[x] Support for deploying multiple instances of the Wso2Is CRD with clustering
[x] Major refactoring of the source code following the best practices of Go
[x] Fixing the delays when applying changes to the custom resources
[x] Added kustomize-supported configurations for creating test MySQL instances

@Mirage20 could you please review this PR

@Mirage20 I did the suggested changes. Could you please check now?


fix money = 1 typo in introduction example

Hi,
In this section of https://www.learntla.com/introduction/example/ page, it should be money = -1, I think.

This is my first ever pull request on GitHub so pardon my mistakes.

Oh I think it's that "when money starts as 1". I'll unify evrything

Closing because archiving in favor of new version


Added google call creds/per_rpc interop tests

This is a breakout of the per_rpc_creds from #6155.
Adds a wrapper for google creds, as well as implementing the per_rpc interop tests in issue #4389

LGTM after fixing the comment format I mentioned.

I've addressed all the comments.

I've addressed the comments.

I've addressed the feedback.

test this please

test this please

test this please

I've addressed the feedback, and also fixed a bug where MetadataPlugin destruction was causing a intermittent segfaults, which caused the interop tests to fail.

Looks good to me apart from two comments; @soltanmm please also review.

Made suggested changes and rebased.

Luhfullyguhtum.

@soltanmm Friendly ping

lgtm on tests

test this please

All relevant tests have passed.


wxGTK 3.0: columns are right-aligned instead of left-aligned

Brazilian Portuguese (pt_BR) uses left-to-right text, which currently doesn't happen. Also, I'd expect 'ID' column to be in the left side (not sure if that's intentional). Please let me know if you need any (and which) debug info.

Basic, always required information is specification of the build, platform, version, all involved files, step by step instructions to reproduce.
I very much doubt this is about pt_BR specifically; I rather suspect another wxGTK issue. Please verify, i.e. try a) running under different locales and b) opening files in different languages.
This doesn't appear to be RTL, by the way — the text isn't rendered the way it would in that case.

Basic, always required information is specification of the build, platform, version, all involved files, step by step instructions to reproduce.

Distro: Arch Linux 64-bit
Package versions:
poedit 2.0
wxgtk 3.1.0 (built with GTK+2)
Steps to reproduce: Open existing PO file, or create from POT
Files involved: Any PO file I opened so far in this version of poedit

I very much doubt this is about pt_BR specifically; I rather suspect another wxGTK issue. Please verify, i.e. try a) running under different locales and b) opening files in different language

You're right. I opened about half of the locales available for poedit and all of them have the same text alignment as pt_BR
I tested the following locales in UTF8: an_ES ar_BH az_AZ be_BY be_BY@latin bg_BG bs_BA ca_ES cs_CZ da_DK de_DE el_GR en_GB en_US es_ES et_EE eu_ES fa_IR fi_FI fr_FR ga_IE gl_ES he_IL hr_HR hu_HU hy_AM id_ID is_IS it_IT ja_JP ko_KR pt_BR uz_UZ vi_VN zh_CN zh_TW

This doesn't appear to be RTL, by the way — the text isn't rendered the way it would in that case.

Right, sorry. What I meant is the text right-aligned for msgid and msgstr (e.g. the highlighted text)

I tested the following locales in UTF8: an_ES ar_BH az_AZ be_BY be_BY@latin bg_BG bs_BA ca_ES cs_CZ da_DK de_DE el_GR en_GB en_US es_ES et_EE eu_ES fa_IR fi_FI fr_FR ga_IE gl_ES he_IL hr_HR hu_HU hy_AM id_ID is_IS it_IT ja_JP ko_KR pt_BR uz_UZ vi_VN zh_CN zh_TW

Thanks for confirming — but wow, talk about thorough testing. Thanks!

Right, sorry.

No need at all, I was just clarifying that, more to myself/record really, to avoid going a wrong path looking for the cause.

The bug is fixed (in wxGTK-3.0.x) in https://github.com/wxWidgets/wxWidgets/commit/ed8c3feab389493429c0a9bdde832b51a1c2186a
I don't see a way to work around it in Poedit code just yet.

I'm sorry, I misread you, I now see that you say this is with 3.1.0 — I can indeed not reproduce with 3.0.2.
If you can reproduce with 3.0.2, I'll try to find some workaround. But since this is fixed in wx master (i.e. what supersedes 3.1.0 unstable release) as well as 3.0 branch (to be 3.0.3 if it ever happens), there's little point in spending time on this — if, again, it doesn't exhibit itself with 3.0.2.

I didn't mention wxgtk 3.0.2 (which is the current version in provided in Arch Linux) because I'm not able to use poedit 2.0.0 built against it as commented on #366 - just to remind you that I built 3.1.0 locally to test poedit.

(With latest Poedit master you should be able to build against wxGTK 3.0.2 and check this behavior too,  if you want (after ignoring the one unrelated assert from #366). But since that assert needs a wxGTK patch to fix anyway, perhaps it doesn't matter, as the above patch could be applied as well.)

Closing because it seems to only affect an old unstable wx release.

Not that old, it's on the last ubuntu release. Is there something I can do to fix it on my ubuntu desktop?

@JSteunou No, it is old, and it is not true that Ubuntu packages unstable wxGTK series. I think you didn't read the content of this issue carefully enough and confuse this with 3.0.2 — for which there is a separate bug, linked just above, with all the explanations you may need.

issue #366 is about poedit not working at all, my issue is about poedit having all the thing right align like @rffontenelle snapshot
I think I'm on the correct issue.
I dont know which version of wxGTK I have, I know I have Ubuntu 17.04, but as it's seems to be the same bug and 17.04 is the latest Ubuntu release, I am surprised this issue is closed because of old unstable release. Is Ubuntu packager used an old unstable release? Can I update it or do something to solve this issue on my desktop?

I am surprised

That would be because you didn't actually read this issue — yet you are so certain you're right despite not even having any clue about the primary aspect, i.e. which wxGTK version you have. Either wait for packaged version of Poedit or use Poedit 2.0.2, which has correct dependencies to avoid both this and #366. In any case, please stop adding any more comment noise to this issue.

I was just asking nicely a little help about how I can fix this bug on my desktop (because this is the bug I have and every Ubuntu user have), as this issue is closed, hence resolve... but yeah I will just stop adding any more noise.


Vafnar's Weekly Record 3/3/17

All ideas were taken from the #design_chat from the discord server.
A) Shell Game Boss
B) Slot Machine Item/Ability that drains money for damage/healing
C) Addons for heroes/bosses like Gyrocopter's Aghnahim scepter that when the hero is killed drops the addon as a "consumable" item that is put onto the hero
D) Boss more powerful during Even/Odd minutes
E) Item that allows ganking by going under the map allowing passage through unpassable places
F) Dragon Mini-boss in order to get to large boss
G) Item that burns like radiance, but the closer you get to them the less it hurts
H) A dodge roll button
Suggestions from @zelding @warpdragon @Kartrix @Haganeko @chrisinajar Myself @Baumi

I posited my problems with A - feels like a luck based encounter. Same with B.
E sounds like a smoke.
I think G is already in? It was proposed some weeks ago, at least.
H is like the dash core thing Baum proposed. We need to remember this is not DS.

I want to add some of my opinions as well:
D seems a bit forced. Those kinds of see-saw power struggle already exists in dota due to heroes being inherently imbalanced during different phases of the game. I don't know about aa2, but I don't like the idea of enforcing a certain rhythm to a game, hope that made sense.
F doesn't sound interesting, it's just 2 bosses.

H sounds like flash in League, and there also evryone take it ;)


Wrong background colors for git status

I'm using Gruvbox colorscheme and for some reason part of git statuses displayed with wrong bg-color:

When I'm trying to modify them via
vim.api.nvim_set_hl(0, 'NeoTreeGitAdded', {bg='#282828'})
vim.api.nvim_set_hl(0, 'NeoTreeGitConflict', {bg='#282828'})
vim.api.nvim_set_hl(0, 'NeoTreeGitDeleted', {bg='#282828'})
vim.api.nvim_set_hl(0, 'NeoTreeGitIgnored', {bg='#282828'})
vim.api.nvim_set_hl(0, 'NeoTreeGitModified', {bg='#282828'})
vim.api.nvim_set_hl(0, 'NeoTreeGitUntracked', {bg='#282828'})

it behaves weird on selection active file:

Could you advice me any options how could I handle this?

Hi! You can do
vim.cmd([[
highlight NeoTreeGitAdded bg=none,
highlight NeoTreeGitConflict bg=none,
...
]])

I'd use vimscript for this instead of lua api to keep the other attributes (e.g fg, gui, ...)

@nhat-vo Doesn't work for me. I.e. the results still same for some reason

Where do you put these lines? You should put them at the end of your configuration

@nhat-vo Here is how it looks for me.
P.S. Moving colorscheme at the bottom did not change things
The neo-tree is lazy-loaded:

Could you try putting
vim.cmd([[
highlight NeoTreeGitAdded bg=none,
highlight NeoTreeGitConflict bg=none,
...
]])

at the end of your init.lua file? Neo-tree set those highlight groups on load, so what you wrote in the gruvbox's init will be overwritten.

Still same result

Do you mean same result as in the original behavior (the first image that you sent)?
If so, did you try doing :highlight NeoTreeGitUntracked bg=none etc. in nvim and see if it changes anything?
If this still does not fix your issue, could you send me your config so I could have a better look?

@nhat-vo In that case it says

Try changing bg to guibg or ctermbg instead. One of those should work, depending on how you run nvim.

@nhat-vo This helps, thanks! But the colors are gone.
But, I've investigated it deeper and find out that:
NeoTreeGitModified links to GitSignsChange
GitSignsChange links to GruvboxAquaSign
and GruvboxAquaSign has this config:
    GruvboxAquaSign = config.transparent_mode and { fg = colors.aqua, reverse = config.invert_signs }

So, as a temporary solution, I just enabled transparent mode and set the background of my terminal to the same color as my neovim theme and it fixed the problem.
Anyway, this looks like a bug, but I am not sure this is related to the neo-tree itself. Looks like this is more for colorscheme.

Gruvbox is quite different from many other themes (e.g tokyonight, rose-pine, catppuccin, etc.) in that it use a different background color for signcolumn from the main background. As a result, when used outside the signcolumn, they will be rendered with a different background, resulting in the error as above. I also encountered the same weird behavior with noice.nvim.

You can also add
require("gruvbox").setup({
    overrides = {
        SignColumn = { link = "GruvboxBg0" }
    }
})

which removes the signcolumn color.
I'm closing this issue as solved.


Versioning notifications are on top of versioning panel

The use of Atom's notification system causes the notifications to be on top of the versioning panel.

Disabling versionning panel #205 .
Since atom is doing a git integration in 1.18, we  don't need it anymore.


The 'Preview' feature disappears after reloading SE

Storage Explorer Version: 1.8.1_20190524.1
Platform/OS: Linux Ubuntu/macOS High Sierra/Windows 10
Architecture: ia32/x64
Regression From: Previous release 1.8.1
Steps to reproduce:

Open Storage Explorer -> Reload(Ctrl +R).

Expect  Experience:
The 'Preview' feature shows well.
Actual Experience:
The 'Preview' feature disappears.

More info:
This issue doesn't reproduce when closing SE then open it again.

This issue has been fixed on latest build_20190524.5 of feature branch. So Close it.


Fix Tesoro-y-Gloria-2E sheet

Submission Checklist
Required

[x] The pull request title clearly contains the name of the sheet I am editing.
[x] The pull request title clearly states the type of change I am submitting (New Sheet/New Feature/Bugfix/etc.).
[x] The pull request makes changes to files in only one sub-folder.
[x] The pull request does not contain changes to any json files in the translations folder (translation.json is permitted)

Changes / Description
Hey, I'm the developer hired by the creators of the game Tesoro y Gloria (Telmo Arnedo). I realized that Roll20 applies the sheet- prefix to all CSS classes, and this is the reason of https://github.com/Roll20/roll20-character-sheets/issues/11489. This should fix the issue for this sheet. There is a separate PR to fix the sheet "Tesoro-y-Gloria-2E", which has the same problem. Thanks in advance!

Character Sheet Info Roll20 Internal Use only.


please add instagram live requests

Please add the methods related to creating a live stream, starting a live stream, and ending a live stream to the library. Thank you.

This library is migrating to a different library. All the further updates will be implemented in that library. Once it is out, I'll pin the link

The endpoints related to the live stream require camera support and captured data. Also, I don't see any practical application of those outside of official client. Hence they will not be implemented


Access-Control-Max-Age header

Here it's the changes that introduce management of the header Access-Control-Max-Age.
I'm still missing some parsing internals, but I think it could be a good starting point.
The reference issue is #5010

I saw that the CI build failed. However, the test that didn't pass doesn't seem to depend on the code I pushed. I supposed there is some non-deterministic test that exceeded the timeout.

Hey, @rossabaker. Sorry for the ping. I want to know if there is any problem with the PR. It's my first contribution, and I probably made something wrong 😄.
Let me know.

I closed this PR since I just opened PR #5171 to the right branch.


RemovedInDjango40Warning

Using this library in Django 3.0+ yields a few RemovedInDjango40Warning warnings:
RemovedInDjango40Warning: django.utils.translation.ugettext_lazy() is deprecated in favor of django.utils.translation.gettext_lazy().
  verbose_name = _("Password Reset Token")
....

This has finally been fixed in #83

Hi, im installing the package with pip install django-rest-passwordreset but this warning still appearing, the package code has the .ugettext_lazy() method. Any idea of what happen?


CP-12359: dijkstra

From unknown CodePlex user on Monday, 12 January 2009 07:37:07
I've problem with dijkstra and the last changeset. It' s too slow.

Attached two sample applications.
The first (Testgraph_01) is developed with a Changeset before 29945 (I don't remember).
Te second one (Testgraph_02) with the last changeset (30070).

As you can see, the second one It's too much slow and uses over 50% of CPU.

Bye
TestGraph_01.zip
TestGraph_02.zip
Graph.zip

Form unknown CodePlex user on Monday, 12 January 2009 07:37:48
Second attachment!

Form unknown CodePlex user on Monday, 12 January 2009 08:59:45
Please send the graph data as GraphML, it is very independent. Thanks.

Form unknown CodePlex user on Tuesday, 13 January 2009 03:47:56
GraphML Graph Format

Form unknown CodePlex user on Tuesday, 13 January 2009 18:19:32
Are you running the Release version?

Form unknown CodePlex user on Tuesday, 13 January 2009 18:31:27
What are the weights?

Form unknown CodePlex user on Tuesday, 13 January 2009 18:32:01
Which root do you use?

Form unknown CodePlex user on Tuesday, 13 January 2009 21:41:06