Add support for the generation of seamlessly-looping animated textures and closed shapes

Hi @lmas. I found your OpenSimplex Python library and am very pleased by it. Great work. I intend to use your library to drive a turbulence generator for a scientific water tunnel I am designing. The coherent nature of Simplex noise is ideal here. The noise pattern we need should loop seamlessly over time, hence my pull request to incorporate two functions called polar_loop_2D_stack() and double_polar_loop_1D_stack().
These functions can also be very useful to others for e.g. the generation of seamlessly-looping animated textures or closed shapes. Perhaps a friendlier name for the functions would be looping_animated_closed_shape() and looping_animated_texture()? Open for debate.
I'll just point you to \examples folder and the animated gif, 2D and animated gif, 1D to show you what the new functions do. Keeps this message still somewhat short. The inspiration is taken from https://www.youtube.com/watch?v=ZI1dmHv3MeM.
Both functions are fully optimized for numba and can show a progress bar during the noise generation which I find very helpful. I tested that the functions work both with numba and numba-progress in the environment, and without.

I'll cancel my pull request for the moment. I am now working on the generation of seamlessly tileable 2D textures and this makes me reconsider the algorithm behind the polar_loop_2D_stack() and double_polar_loop_1D_stack() functions. I'll be back with a new pull request once I have convinced myself I am happy with the algorithm.
Best, Dennis

Continued in #36


Video config undefined on npx next-video sync with Mux

Hi everyone,
I'm encountering a problem when syncing a video from my /video folder to MUX.
The Problem
This is the error when I execute the command: npx next-video sync
+ Found 1 unprocessed video
- Processing file: landing.mp4
✗ An unknown error occurred TypeError: Cannot read properties of undefined (reading 'provider')
    at createAsset (file:///Users/carpi/Desktop/Clan/clanid.it/node_modules/next-video/dist/assets.js:40:27)
    at async newFileProcessor (file:///Users/carpi/Desktop/Clan/clanid.it/node_modules/next-video/dist/cli/sync.js:62:24)
    at async Promise.all (index 0)
    at async Object.handler (file:///Users/carpi/Desktop/Clan/clanid.it/node_modules/next-video/dist/cli/sync.js:88:24)

As you can see, the file is found but it fails to read the video configs, as shown into the actual piece of code that fails:
async function createAsset(filePath, assetDetails) {
  const videoConfig = await getVideoConfig(); <-- Here the video configs seems to be undefined
  const assetConfigPath = await getAssetConfigPath(filePath);
  let originalFilePath = filePath;
  if (!isRemote(filePath)) {
    originalFilePath = path.relative(cwd(), filePath);
  }
  const newAssetDetails = {
    status: "pending",
    // overwritable
    ...assetDetails,
    originalFilePath,
    provider: videoConfig.provider, <-- here's the line that throws the error
    providerMetadata: {},
    createdAt: Date.now(),
    updatedAt: Date.now()
  };
  [[ other code ]]
}

My setup
NextJS version: 14.0.4
Next Video version: 1.0.4
Language: TypeScript
video.d.ts file: ✅ Created
tsconfig.json file: ✅ Modified with video.d.ts included
next.config.mjsfile: ✅ Modified
On the next.config.mjsfile I need to point out the fact that since my project is using MDX I did have to combine the configurations for Next Video in a way that can work with the already present ones for MDX support.
import rehypeShiki from '@leafac/rehype-shiki'
import nextMDX from '@next/mdx'
import { Parser } from 'acorn'
import jsx from 'acorn-jsx'
import escapeStringRegexp from 'escape-string-regexp'
import * as path from 'path'
import { recmaImportImages } from 'recma-import-images'
import remarkGfm from 'remark-gfm'
import { remarkRehypeWrap } from 'remark-rehype-wrap'
import remarkUnwrapImages from 'remark-unwrap-images'
import shiki from 'shiki'
import { unifiedConditional } from 'unified-conditional'
import { withNextVideo } from 'next-video/process'; <<< NEXT VIDEO CHANGES

/** @type {import('next').NextConfig} */
const nextConfig = {
  pageExtensions: ['js', 'jsx', 'ts', 'tsx', 'mdx'],
}

function remarkMDXLayout(source, metaName) {
  let parser = Parser.extend(jsx())
  let parseOptions = { ecmaVersion: 'latest', sourceType: 'module' }

  return (tree) => {
    let imp = `import _Layout from '${source}'`
    let exp = `export default function Layout(props) {
      return <_Layout {...props} ${metaName}={${metaName}} />
    }`

    tree.children.push(
      {
        type: 'mdxjsEsm',
        value: imp,
        data: { estree: parser.parse(imp, parseOptions) },
      },
      {
        type: 'mdxjsEsm',
        value: exp,
        data: { estree: parser.parse(exp, parseOptions) },
      },
    )
  }
}

export default async function config() {
  let highlighter = await shiki.getHighlighter({
    theme: 'css-variables',
  })

  let withMDX = nextMDX({
    extension: /\.mdx$/,
    options: {
      recmaPlugins: [recmaImportImages],
      rehypePlugins: [
        [rehypeShiki, { highlighter }],
        [
          remarkRehypeWrap,
          {
            node: { type: 'mdxJsxFlowElement', name: 'Typography' },
            start: ':root > :not(mdxJsxFlowElement)',
            end: ':root > mdxJsxFlowElement',
          },
        ],
      ],
      remarkPlugins: [
        remarkGfm,
        remarkUnwrapImages,
        [
          unifiedConditional,
          [
            new RegExp(`^${escapeStringRegexp(path.resolve('src/app/blog'))}`),
            [[remarkMDXLayout, '@/app/blog/wrapper', 'article']],
          ],
          [
            new RegExp(`^${escapeStringRegexp(path.resolve('src/app/stories'))}`),
            [[remarkMDXLayout, '@/app/stories/wrapper', 'story']],
          ],
        ],
      ],
    },
  })

  return withMDX(withNextVideo(nextConfig)); <<< NEXT VIDEO CHANGES
}

Conclusion
I'm trying to understand where's my error in the implementation of the library. I was able to get it working (kinda) using a file already present on MUX, but now I want to have also the sync functionality working.
Thanks!

be sure to also set the folder property in the next-video config!
this is not very well documented I see.
https://github.com/muxinc/next-video/blob/a8db8244a3aa0f7ce220ebf97f0050d3569d4233/src/config.ts#L9-L27
// export default withNextVideo(nextConfig, {
//   folder: 'video'
// });

Thank you @luwes very much!
I didn't expect a response and even a new release of the library in just 6 hours, awesome!

Sorry for the delay @luwes

is it on Windows or Mac?

Linux

what is your module resolution in tsconfig?

"moduleResolution": "node"

The problem with my next.config.js was in the webpack configuration. After I fixed it, the undefined provider error no longer occurred, but a new one was returned:
✗ An unknown error occurred MuxError: The MUX_TOKEN_ID environment variable is missing or empty; either provide it, or instantiate the Mux client with an tokenId option, like new Mux({ tokenId: 'my token id' }).
    at new Mux (.../node_modules/@mux/mux-node/index.mjs:29:19)
    at initMux (.../node_modules/next-video/dist/providers/mux/provider.js:11:17)
    at Module.uploadLocalFile (.../node_modules/next-video/dist/providers/mux/provider.js:99:3)
    at uploadLocalFile (.../node_modules/next-video/dist/handlers/local-upload.js:6:23)
    at callHandler (.../node_modules/next-video/dist/video-handler.js:11:10)
    at newFileProcessor (.../webapp/node_modules/next-video/dist/cli/sync.js:65:16)
    at async Promise.all (index 0)
    at async Object.handler (.../node_modules/next-video/dist/cli/sync.js:88:24)

I'm properly setting the MUX_TOKEN_ID and MUX_TOKEN_SECRET in my .env.local. So I don't know why this error is returned.

@pedrodalmas did you find the issue of the missing env vars?
this is very hard to debug without reproducing it so if you could make a minimal reproduction online.
https://github.com/muxinc/next-video/issues/251#issuecomment-2112789735

closing due to inactivity


[BUG][opensearch + opensearch-dashboards]

Describe the bug
So I'm trying to deploy a "mostly" vanilla deployment of these two charts but I'm getting an error in the dashboard logs.
{"type":"log","@timestamp":"2022-10-28T15:02:05Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: socket hang up"}
{"type":"log","@timestamp":"2022-10-28T15:02:07Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: socket hang up"}

As far as I can tell the logs in the opensearch cluster pods look good.
To Reproduce
I'm running this in RKE2 but I'm granting unrestricted access via podSecurityPolicies.
helm install opensearch opensearch/opensearch -n admin -f values.yaml
values.yaml:
rbac:
  serviceAccountName: "admin"
podSecurityPolicy:
  create: false
  name: "system-unrestricted-psp"

helm install opensearch-dashboards opensearch/opensearch-dashboards -n admin -f dashboard-values.yaml
dashboard-values.yaml:
serviceAccount:
  name: "admin"
extraInitContainers:
- name: increase-vm-max-map
  image: busybox
  command: ["sysctl", "-w", "vm.max_map_count=262144"]
  securityContext:
    privileged: true

I can provide full log output if necessary, the only other thing I see that seems odd is this log the opensearch cluster pods:
tee: /usr/share/opensearch/config/opensearch.yml: Permission denied

Expected behavior
I'm expecting that the dashboard can connect to the opensearch cluster automatically
Chart Name
opensearch and opensearch-dashboards
Host/Environment (please complete the following information):
helm version
version.BuildInfo{Version:"v3.8.2", GitCommit:"6e3701edea09e5d55a8ca2aae03a68917630e91b", GitTreeState:"clean", GoVersion:"go1.17.5"}

Kubernetes Version: [v1.22.6+rke2r1]

Additional context
Add any other context about the problem here.

Hey @kevinlmadison does the default installation work without modifying the security context ?
Thanks you

Hey @kevinlmadison default installation works fine without any issues.
I did notice you added podSecurityPolicy for  opensearch  and set to false and still declared the system-unrestricted-psp, I assume this wont create any PodSecurityPolicy.
For opensearch-dashboards, I have tested adding increase-vm-max-map and works fine
k logs osd-opensearch-dashboards-59bfd4c85-jkkjh  -c increase-vm-max-map
vm.max_map_count = 262144

Do you still encounter this issue? If so I suspect it has to do with your underlying disk, can you check if the mount has write permissions?
Adding @TheAlgo @bbarani @peterzhuamazon


WIP: displaying admins

What's the bitwise stuff for

What's the bitwise stuff for

There is a single integer database column.  As the "permissions" expand, that same field can be re-used over an over.   For example:
1 = Manage Users
2 = Manage Service Bodies
4 = View Only Service Bodies
8 = Manage Configuration.
These can then be summed together.  So let's say the user has the ability to Manage Service Bodies + Manage Configuration.  The sum would be 10.
On the way out, you can decompose that to know which settings the user has.

ahh cool

this looks pretty good, I can test more tomorrow.

self password reset doesnt seem to be working, If i log in as a non admin and go to my profile and change password, hit save and log out. the new password does not work. i can see the hash in DB never changed either.

self password reset doesnt seem to be working, If i log in as a non admin and go to my profile and change password, hit save and log out. the new password does not work. i can see the hash in DB never changed either.

fixed


multiPlayer not working

I am trying to use the multiPlayer  but it doesn't seem to work any ideas as to what I am missing here is the link.
https://codesandbox.io/s/naughty-tdd-9mzl9

@tambien  is there a way to call start on Players? and start all tracks at once? Or what is the right way , I have multiple tracks ( like drum , guitar etc) I want to play them together and do some mixing how can I achieve it?


Generate types inside a theme package

When developing a theme, types are generated inside a playground. This makes it hard to properly link types while developing a theme. To improve this, types could also be generated inside the theme package to prevent issues with hoisting or circular type references.
Things to consider:

Types are generated using the AIK utility addDts. This utility only generates types inside a user project, this issue would require a custom utility (or an upstream addition to AIK) for type generation inside a package
Where are these types generated inside a package? virtual.d.ts?

I am going to take over this issue and rename it, I worded it poorly and didn't explain exactly why I created it. Generating types inside the theme's package may have some advantages, but I don't know of any right now.
The real issue here is that theme authors need Content Collections types while developing inside a package.
This is a bit complicated because we need to generate the Content Collection types inside a playground and hoist them somehow so that authors have type safety when creating theme packages.
TLDR: Typically, types flow downwards from packages to users. We need to pass types upwards from a user to a package.


What is the equivalent for batch overfitting for such a training scheme?

I've the following understanding:
The idea is to see if the Generator, when trained exclusively on the small batch, can produce samples that the Discriminator thinks are real. If your network setup and loss functions are working correctly, the Discriminator should become uncertain about whether the samples are real or fake (i.e., its loss should hover around the value indicating a 50% guess). The Generator's loss should decrease, showing it's generating better samples.
How to achieve this quickly to ensure everything in training is setup properly. I'm not getting such a behavior.

I've the following understanding:
The idea is to see if the Generator, when trained exclusively on the small batch, can produce samples that the Discriminator thinks are real. If your network setup and loss functions are working correctly, the Discriminator should become uncertain about whether the samples are real or fake (i.e., its loss should hover around the value indicating a 50% guess). The Generator's loss should decrease, showing it's generating better samples.
How to achieve this quickly to ensure everything in training is setup properly. I'm not getting such a behavior.

The adversarial loss in our design is used along with other loss terms (reconstruction loss, cyclic consistency loss) to optimize the model parameters during the training process, so it is difficult for the output of the discriminator to hover around the value indicating a 50% guess as GAN-based image generation methods which only use a single adversarial loss when training.  However, the adversarial loss still plays a role by forcing the retargeted motion to fall into the target motion manifold.


discord webhook pfp

Describe the feature
i know its small but anyway you could set the profile picture of the webhook for panic buttons and all that to the cad logo
Additional Context
No response

This is now be supported. It could be that you have to update the CAD-settings' webhooks page for it to update the avatars


Add GET api/customers/me endpoint to return CustomerDto of authenticated user

Adding GetCustomerByGuid method to CustomerApiService is needed.

Added in 0b887b0d


Feature request .   Saving design json in file.

Can't we provide a functionality to save the json design in file instead of the console.

@vikaskyadav When you call unlayer.saveDesign, it will return the json in a callback function. You can do whatever you want with it.


Update WordPress to 5.2

WordPress 5.2 is out 🎉 This PR will bump the installed version to 5.2.
Notes
Wordpress 5.2 supports PHP 7.3. Do you want to wait until PHP 7.3 is available in the default apk repositories or should we find a more intrusive way to update PHP?
Cheers!

Thanks man! At this moment I want to wait for PHP 7.3 becomes available in the default APK repo's. Too bad that it already takes some time to get it there.


bot is down

Describe the bug
wont let me caption or deepfry images
Steps to reproduce

type any command
send command once you’re done typing it
congrats it gave you the same error message I’m getting

Expected behavior
pretty sure most people know how it’s supposed to work?
Screenshots

Self-hosted instance?
No
Additional context
No response

Duplicate of #383


[🔨 style]: consider adding number-leading-zero rule to stylelint config

As an ESL contributor, I recomend to consider adding number-leading-zero rule to stylelint config.
As for right now we use both options with leading zero and without leading zero.
Expect the style guide to be consistent across project files.
Expected style:

no leading zero .01em

Originally posted by @dshovchko in https://github.com/exadel-inc/esl/pull/1119#discussion_r928635379

:tada: This issue has been resolved in version 4.0.0-beta.18 :tada:
The release is available on:

npm package (@beta dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This issue has been resolved in version 4.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


MBS 0111 Variant

What is the context of this PR?
Added the Blocks and Manifest to build mci_0111
How to review
Run it and review if it matches the Google Slides.
https://docs.google.com/presentation/d/1yoo1dFRqe4XeBArDqIY_vg5G6LeDt_6W7iyW60Aj2Fg

Noticed code refers to mci and RSI but its for MBS
Title in heading - lose RSI
Reporting period

can we get the date to stay together on 1 line, as currently wraps after day and the rest of date on second line
there isnt any validation on radio options, so if you dont select an option it takes you to change the date

Date change

doesnt validate if the 'to' date being before the 'from' date

validation messages - can we use new error messages? - slides are being updated for other MBS forms types https://digitaleq.atlassian.net/wiki/spaces/EQ/pages/110854206/Error+messages+and+standard+text
summary
for radio options can we remove the bullet point

highlighting shouldn't be on trailing comma ,

Title shouldn't include "Retail Sales Index"

Got an error submitting with a comment:
eq-survey-runner_1     | Traceback (most recent call last):
eq-survey-runner_1     |   File "/usr/local/lib/python3.4/site-packages/flask/app.py", line 1475, in full_dispatch_request
eq-survey-runner_1     |     rv = self.dispatch_request()
eq-survey-runner_1     |   File "/usr/local/lib/python3.4/site-packages/flask/app.py", line 1461, in dispatch_request
eq-survey-runner_1     |     return self.view_functions[rule.endpoint](**req.view_args)
eq-survey-runner_1     |   File "/usr/local/lib/python3.4/site-packages/flask_login.py", line 792, in decorated_view
eq-survey-runner_1     |     return func(*args, **kwargs)
eq-survey-runner_1     |   File "/usr/src/app/app/views/questionnaire.py", line 120, in post_block
eq-survey-runner_1     |     return submit_answers(eq_id, form_type, collection_id, metadata, answer_store)
eq-survey-runner_1     |   File "/usr/src/app/app/views/questionnaire.py", line 228, in submit_answers
eq-survey-runner_1     |     message = convert_answers(metadata, g.schema_json, answer_store, path_finder.get_routing_path())
eq-survey-runner_1     |   File "/usr/src/app/app/submitter/converter.py", line 65, in convert_answers
eq-survey-runner_1     |     payload['data'] = convert_answers_to_data(answer_store, questionnaire_json, routing_path)
eq-survey-runner_1     |   File "/usr/src/app/app/submitter/converter.py", line 97, in convert_answers_to_data
eq-survey-runner_1     |     answer_data = _get_answer_data(data, answer_schema['q_code'], value)
eq-survey-runner_1     | KeyError: 'q_code'

error message is wrong (see spec)

Error messages don't match spec

Error message content isn't to spec

Is there a way we can stop this wrapping mid-word on the summary screen (applicable too all content, not just this particular answer)?

Display of dates aren't consistent here, sometimes it has '01 May' and others just '1 May', I checked with Sarah and Alison, it should always be '1 May' without any leading zeros.


fix incorrect encoding of urls in finding list headings

fixes https://github.com/DefectDojo/django-DefectDojo/issues/5070

Not sure if this is the best way. I always learned that when building querystrings, every value should be url encoded seperately and only after that construct the querystring. But that would leave us still with "&not_...." causing problems. So escaping the whole things seems like the easy way out.


Create Posts “sadness-i-thought-will-never-leave”

Automatically generated by Netlify CMS

👷 Deploy Preview for benevolent-pika-f5fbe0 processing.

Name
Link

🔨 Latest commit
fbf128c2e62ddb4c0fe6bd895c7f9b537354b6cc

🔍 Latest deploy log
https://app.netlify.com/sites/benevolent-pika-f5fbe0/deploys/62e3a2b82c526000084cd638


FreeRTOS+TCP: Let xDataLength always mean "total number of bytes"

Description
A network buffer (NetworkBufferDescriptor_t) has a field xDataLength. This is the total length of the buffer. It includes the 14-byte Ethernet header and everything after that.
Just recently, @aggarg  wanted to add a length check in the function vARPGenerateRequestPacket(), something like:
	configASSERT( pxNetworkBuffer->xDataLength >= sizeof(ARPPacket_t) )

This function failed because there are two cases in which xDataLength has a different meaning.

In an outgoing ICMP/ping packet, it means the number of bytes to send plus the size of an ICMP header.
In an outgoing UDP packet, it is the payload size ( without the UDP header ).

In order to avoid confusion, I'd like to correct that: xDataLength will always contain the total number of bytes.
Checklist:

[x] I have tested my changes. No regression in existing tests.
[ ] My code is Linted.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/bot run checks

When UDP packets are sent or received, the field xDataLength still contained the size of the nett payload. I believe that this was the ultimate situation in which xDataLength referred to a payload size.
A simple change in FreeRTOS_Sockets.c in FreeRTOS_recvfrom():
-	lReturn = ( int32_t ) pxNetworkBuffer->xDataLength;
+	lReturn = ( int32_t ) pxNetworkBuffer->xDataLength - sizeof( UDPPacket_t );

In FreeRTOS_sendto() there is this change:
-	pxNetworkBuffer->xDataLength = xTotalDataLength;
+	pxNetworkBuffer->xDataLength = xTotalDataLength + sizeof( UDPPacket_t );

In FreeRTOS_DNS.c, added a variable uxPayloadSize in a few occasions:
-	if( pxNetworkBuffer->xDataLength >= sizeof( DNSMessage_t ) )
+	size_t uxPayloadSize = pxNetworkBuffer->xDataLength - sizeof( UDPPacket_t );
+	if( uxPayloadSize >= sizeof( DNSMessage_t ) )

In FreeRTOS_IP.c, there is an extra sanity check on the UDP field usLength:
if ( ... && FreeRTOS_ntohs( pxUDPPacket->xUDPHeader.usLength ) >= sizeof( UDPHeader_t ) )

All tested again with and without: DNS, LLMNR, NBNS, and UDP messages in both directions.
I also check the values of variables by stepping through the code.

Zero-copy: it returns a pointer to the Network Buffer at the offset of the payload.

The user receives an object of type void * which can be read from ( character bytes ), and which shall be released by calling :
    void FreeRTOS_ReleaseUDPPayloadBuffer( void *pvBuffer )

We don't know in advance if ``xDataLength` is correct. Therefor we take the smaller of the 2 calculated payloads, with a minimum of 0 bytes.
( not sure if a zero-length UDP-packet has any use in real life ).
Hein

/bot run checks

Since we got two approvals and tests are passing, I'm merging this PR as part of the oncall duty.


A little compatibility issue with ReservedFlashlightSlot v1.6.2

A laser pointer will control turrets only if the left alt is held

Hey, I could not understand if its a suggestion or if its an issue..? can you share more information about it

you can make reserved slot be a toggle instead of hold go to core config and check focused slots to true

If I pick up a laser pointer while playing with ReservedFlashlightSlot it will be placed in the reserved slot. I can turn on the pointer but it will not control turrets.

But I can hold the left alt, turn on the pointer and it will control turrets

So I can use both mods, but it looks like it shouldn't be the way it is. That's all, just wanted to notify you.

By the way, because of this I even thought at first that your mod doesn't work. But then I eventually realized what I need to do.

Hey @Multiname thanks for the further information. I will look into it and I will work on a solution for that.

Found the bug and successfully reproduced it. Working on it.

Hey @Multiname and @sword11247. Issue fixed and it will be available in the next release - v0.1.4. Release schedule for Sunday, February 4th


[SPARK-16101][SQL] Refactoring CSV read path to be consistent with JSON data source

What changes were proposed in this pull request?
This PR refactors CSV read path to be consistent with JSON data source. It makes the methods in classes have consistent arguments with JSON ones.
UnivocityParser and JacksonParser
private[csv] class UnivocityParser(
    schema: StructType,
    requiredSchema: StructType,
    options: CSVOptions) extends Logging {
  ...

def parse(input: String): Seq[InternalRow] = {
  ...

class JacksonParser(
    schema: StructType,
    columnNameOfCorruptRecord: String,
    options: JSONOptions) extends Logging {
  ...

def parse(input: String): Option[InternalRow] = {
  ...

These allow parsing an iterator (String to InternalRow) as below for both JSON and CSV:
iter.flatMap(parser.parse)

How was this patch tested?
Existing tests should cover this.

Test build #71770 has started for PR 16669 at commit 28258f7.

cc @cloud-fan, I tried to only change the parsing path (not schema inference or de-duplicating other filtering logics). Could you take a look and see if it makes sense?

Test build #71770 has finished for PR 16669 at commit 28258f7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71770/
Test PASSed.

Test build #71783 has started for PR 16669 at commit b2938ae.

Test build #71783 has finished for PR 16669 at commit b2938ae.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71783/
Test FAILed.

retest this please

Test build #71789 has started for PR 16669 at commit b2938ae.

Test build #71789 has finished for PR 16669 at commit b2938ae.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71789/
Test PASSed.

thanks, merging to master!

I think this PR has broken the 2.10 build.
https://amplab.cs.berkeley.edu/jenkins/view/Spark QA Compile/job/spark-master-compile-sbt-scala-2.10/3619/
Can you please fix it?

Oh, sure. I think I should fix the default scala version to 2.10 in my local..

Let me know when you have a PR to fix it.

Sure, I will send a PR as soon as after building it with 2.10.


Match Speed

Our match speed is lacking compared to the original version, on the order of about 6-8M operations/s.
My changes to construction of OneOf's in my outstanding PR however, is a huge improvement over it, on the order of 1.5-4x depending on the case.
Let's figure out a way to fix that.
Ideas:
Some way to build the matching functionality in a way that doesn't allocate multiple objects, and that can terminate the fall through flow early.  Much more in a builder fashion than we have now.
Perhaps be able to pool them behind the scenes on a per matching functionality (matching arity with the type of the function).  This could also allow a reified optimized matcher that could be used over a number of particular OneOfs by the consumer.

Gidday James
I could be wrong, but as the OneOfMatcher and Switcher are both structs, there should be near nil overhead in their construction.  I suspect the bigger cost is all of the nested function calls.
Time to crack open a profiler I think!

I'd say that's probably a good idea.  I'll see about checking that out in the next couple weeks.  I'll probably refactor the perf tests into unit tests that the Main method calls so that I can run them in isolation with dotTrace.

I changed OneOfMatcher back to being a builder (that was our original version) and sped up about 10%.  I then changed it to a class and it jumped up from 12m to 27m.    The builder loves being a class because it's faster to copy a single reference around rather than the 3 members of the struct.  Before you ask, the arity version runs slower as a class, I checked.
I also implemented Match where it just returned a value without needing a func, eg Match("it's a C1") , and that hit approx 16m.  The reason it's not as fast the the Func version is that it needs to call GetBestType to determine if the X in Match(TResult result) matches any of the Tn's, whereas the Func version  knows at compile time and can save that overhead.  No point keeping this, the Func version is better.
The downsides of being a builder again is that we lose the fancy arity stuff, such as catching missing Match calls at compile-time if you added a new Tn to the OneOf definition.  And we now have to call Result to get the final result.
One step forwards, one step back.  Do you want me to commit it in?  That'll mean bye bye arity but hello speed.  Thoughts?

Can you throw it on a branch? I'll check it out tonight.
On Feb 12, 2017 1:03 AM, "Will Mooar"<EMAIL_ADDRESS>wrote:

I changed OneOfMatcher back to being a builder (that was our original
version) and sped up about 10%. I then changed it to a class and it jumped
up from 12m to 27m. The builder loves being a class because it's faster to
copy a single reference around rather than the 3 members of the struct.
Before you ask, the arity version runs slower as a class, I checked.
I also implemented Match where it just returned a value without needing a
func, eg Match("it's a C1") , and that hit approx 16m. The reason it's not
as fast the the Func version is that it needs to call GetBestType to
determine if the X in Match(TResult result) matches any of the Tn's,
whereas the Func version knows at compile time and can save that overhead.
No point keeping this, the Func version is better.
The downsides of being a builder again is that we lose the fancy arity
stuff, such as catching missing Match calls at compile-time if you added a
new Tn to the OneOf definition. And we now have to call Result to get the
final result.
One step forwards, one step back. Do you want me to commit it in? That'll
mean bye bye arity but hello speed. Thoughts?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Jagged/OneOf/issues/13#issuecomment-279199106, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AD_ohZCLyrk3G2W0wrgm8BxHhkiQpBJIks5rbqCegaJpZM4LzpjC
.

I think I'm going to try adding the interfaces to the matcher. Well require sum(0,n) match methods on the matcher, where n is the max number of Tn we support. And we will require the sum of the nth row of Pascal's triangle of interfaces on the matcher.  This should not have a ton of overhead, and I think we discussed it before. There may be a performance degradation there may be the execution of the method as an interface rather than as the object directly.

Alright, I had to do some pretty tricky stuff to get this working, but I eked out somewhere between a 20 and 100% performance increase on the matcher.  I got fairly varying, but consistently higher results in my tests.
Match1 ... 11.64M per second
Match5 ... 19.03M per second
Match9 ... 16.22M per second
Match11 ... 16.87M per second
Match15 ... 16.72M per second
Match19 ... 15.56M per second
Match1 ... 18.34M per second
Match5 ... 23.74M per second
Match9 ... 20.07M per second
Match11 ... 19.86M per second
Match15 ... 19.74M per second
Match19 ... 20.08M per second
Check out https://github.com/jamesbascle/OneOf/commit/07f81570d023b8b49d14967e88a4af560e98833c
The general strategy is to create a whole mess of interfaces, each one representing one piece of a giant state machine. Each one represents the remaining options for matching, and the result of each of those matches.  These interfaces are all implemented, explicitly, by the matcher, to keep from having any ambiguities.
This is just a proof of concept... I think I'm going to give the code base some loving care before I'm ready for this to go out.  For that I'm going to make a new issue describing my plans.

I haven't read the code yet, but just the thought turns my head inside out!  Well done!
I'm heading in, send in a search party if I don't surface in a week!

Haha, thanks!  Good luck!
As an aside, 2 tests don't pass at the moment because they were using dynamic references to the one of, and since the match methods are implemented as explicit interface implementations, and you can't use those when you have a direct reference to the object, AND the dynamic keyword runs the code as if the type of the reference at compile time were the type of the reference at runtime, it can't find the match methods.
They are, however, clearly there.  Not sure what to do on that count ATM.

It's an obscene abuse of interfaces, but I'm impressed.  I call that dogged persistence to ek out that last bit of speed while retaining the arity niceties.
I was considering trying to make the syntax close to the new C# 7 switch, and hopefully in time match, if it ever comes to C#.  eg  oo.MatchGiving<int>.Is<string>().And(s => s == "Hello").Then(s => 1).etc .  But this would require a builder style rather than arity, which would take things in a different direction.
ps Just saw your comment re the tests.  I may have to read that several times for it to sink it.  That, or drink more.  ;-)

This compiles and possibly deals with the dynamic binding issue, but creates a new issue:
    interface ISomething
    {
        void X();
    }

    class Z : ISomething
    {
        // ick
        public void X() { ((ISomething)this).X(); }
        void ISomething.X() { ... }
    }

I've got the code in a functional state again with the MV pattern for OneOf in OneOf.tt. I also separated out the various entities into their own .tt files for easier modification and side by side work.
I'm most interested in hearing feedbeack on the OneOf.tt changes.  To my eyes the code is far simpler now - the tt just acts as a simple template to plop the variable strings into the OneOf structure. see https://github.com/jamesbascle/OneOf/tree/opt-OneOfMatcher

I knew I should have been worried when you went quiet for a long period!  Things are dramatically simpler in OneOf.tt, and I like how you've still allowed the tt files still be the templates (the view) while using the library to describe the OneOf etc (the model).
I'd be happy to accept a PR, or I just switch to using your fork.  This project has to one of the most over-engineered projects on GitHub!  It's good for learning how far we can push C#, and I have to say you've taught me a thing or two. =)

I've still got a bit of work to do, but i'll have a PR ready in the next couple weeks.  Just wanted to see your thoughts before fully pursing that strategy. I plan to move each of the templates to that sort of paradigm, and change the switcher to use the same strategy as the matcher.
I also think I can get rid of needing a different matcher for each Arity, by filling in some internal sealed empty None class into the last N-1 Parameters and casting it to the [1, n] version of the interfaces when constructing the matcher/switcher from a OneOf.  This will reduce the amount of classes in the code to 1 Matcher, 1 Switcher, and a OneOf for each arity.
As for the over engineered bit - perhaps a little, but It's actually a pretty tough problem to solve in a language that doesn't support them OotB, and to get the cleanest most natural-feeling API for pattern matching.  Once we're moved to a model/view setup, I think we're in a good place for re-evaluating the API as you've mentioned doing in the past - but first things first.

~80m/s constructing, ~20m/s matching
Maybe get things stable then start talking about more states for the state machine?

I consider this pretty much closed.  Going to open up a new issue for the .With() method being added full-fledgedly to the state machine rather than being a bit wonky in terms of the types.
BTW, @Jagged can you send me an email?  I tried emailing you at the email address via which we communicated previously, but I got NXDOMAIN for your domain.


Idea: Pow trait

I've been working on a crate which works with runtime-level units, and I think that having a generic Pow trait would be useful.
Essentially, it would be very similar to the other unary op traits, namely:
pub trait Pow<RHS> {
    type Output;
    fn pow(self, rhs: RHS) -> Self::Output;
}

And in most cases, this would simply do exponentiation by squaring as included in the existing pow functions. However, it may be easier to implement for some types, namely a unit could simply multiply a field that indicates what power of the given unit you have. Another example is that a type which constrains which values it has, e.g. a matrix which specifically performs scaling operations, could be modified to do less work.
This would be weird to implement and may cause confusion with the existing pow function, which is why I haven't offered a PR for it. But it might be nice to have in this crate.

It makes sense to me, especially since types can make internal decisions about stuff like #18.

(We can de-emphasize the generic fn pow() in favor of this.)

It might also be interesting that floats can implement both Pow<Self> and Pow<i32> using powf and powi, though only with std enabled.  All of the integers can forward to their inherent pow methods too, instead of using our own.

Looks good :) I think it is very useful since it can be implemented for num_complex::Complex<T>.


Are our protocols too slow?

Why: Problem description :grey_question:
If I run this file a few times, I get some interesting results.
packages/server-wallet/src on  ags/log-crank-errors [$!+?] via ⬢ v12.16.3 on ☁  (us-west-1) on ☁<EMAIL_ADDRESS>took 14s
❯ ts-node protocols/signer.ts
Add hash x 10000 / useClass: true: 680.431ms
Add hash x 10000 / useClass: false: 657.099ms
constructing a class x 10000: 0.648ms

packages/server-wallet/src on  ags/log-crank-errors [$!+?] via ⬢ v12.16.3 on ☁  (us-west-1) on ☁<EMAIL_ADDRESS>took 12s
❯ ts-node protocols/signer.ts
Add hash x 10000 / useClass: true: 780.812ms
Add hash x 10000 / useClass: false: 654.982ms
constructing a class x 10000: 0.643ms

packages/server-wallet/src on  ags/log-crank-errors [$!+?] via ⬢ v12.16.3 on ☁  (us-west-1) on ☁<EMAIL_ADDRESS>took 12s
❯ ts-node protocols/signer.ts
Add hash x 10000 / useClass: true: 703.148ms
Add hash x 10000 / useClass: false: 658.250ms
constructing a class x 10000: 0.989ms

IE. running code that's an instance method is ~8% slower, in this case, than running equivalent code that's a function. (I have no idea why.)
Note that we use the useClass: true pattern in our objective crankers. EG.

We do ChannelOpener.create which is assigned to this.channelOpener
We only ever call this.channelOpener.* in one place.

This could be replaced with a set of plain old functions in channel-opener, and the following pattern
import * as ChannelOpener from 'path/to/channel-opener';

ChannelOpener.crank(interestingArgs, servicesWithSideEffects)

Note these patterns do not happen on the critical path. Nevertheless, in the context of a "busy" channel manager (eg. the gateway in The Graph Network) the performance of channel management is actually relevant, as the service may become CPU bound.
What: Describe what a solution should resolve :grey_exclamation:

[ ] Create a benchmark to confirm that we can reduce CPU load of channel management
[ ] If the benchmark yields a positive result, plan an epic for refactoring class-instance-method based computation as module-based functions. Create the required cards.

Project management
Before closing this card,

[ ] I have created any required issues or epics
[ ] I have updated the related notion or epic documentation

Is there a specific performance problem that you're tackling here? If so, is there any analysis of that problem that points to this issue as the root cause? It isn't clear why the stated problem - that running a given file gives some interesting results - is a problem that we should be focussed on.
It is also unsurprising to me that instantiating a class and calling an instance method which calls function x is slightly slower than calling function x directly. It doesn't follow that running an instance method is slower than calling a function.

Not a priority to have high throughput for sprint 4-6. Revisit this in sprint 6.


Parser refactoring + Hygienic macro system

@kha @gebner: we have been talking about the parser refactoring for a few months.
@david-christiansen has many ideas and suggestions. I'm open this RFC to record his ideas and collect suggestions.
David is suggesting an architecture where we would have the following main steps:
1- read : string -> syntax
2- expand : syntax -> pre-expr
3- elaborate : pre-expr -> expr
The read procedure would be much simpler than the current lean parser, and it would generate a s-expression like data structure (syntax). It would include precise position information and comments.
Here are some links that David sent me: https://www.cs.indiana.edu/~dyb/pubs/LaSC-5-4-pp295-326.pdf, https://www.cs.utah.edu/plt/publications/popl16-f.pdf
We would be able to reconstruct the input string faithfully from a syntax object produced by the read procedure. The syntax object will also represent faithfully constructs such as match-exprs, do-blocks, calc-blocks, etc which are currently expanded at parsing time. Names would not be resolved by the read procedure. It is not clear to us whether we should combine the expand and elaborate steps or not. It seems it would be a good idea to have both of them at least initially. In this way, the elaborator code would not be affected. The downside of not combining them is that expansion cannot take type information into account.
Users would be able to define new syntax using Lean. David also suggested that we could even have different read procedures like Racket.  Here is a paper describing how this works in Racket: http://users.eecs.northwestern.edu/~stamourv/papers/languages-the-racket-way.pdf
An important aspect of this proposal is that the expansion step is implemented in Lean, which means expanding user notation become far more flexible, and we can define new syntax based on old syntax.
In the current implementation, new syntax must be mapped into pre-terms. So, it is non-trivial to add new syntax that extends existing syntax.
This approach is also useful for implementing tools such as formatters and linters using Lean.
I'm not sure now it is best time for implementing this change, but I believe it would simplify the current implementation and make it much more flexible.

The syntax object will also represent faithfully constructs such as match-exprs, do-blocks, calc-blocks, etc which are currently expanded at parsing time.

Notation should probably also remain unexpanded then.

Is there anything better in the literature? Skimming through the papers that Leo posted, nobody even seems to mention infix operators.

Most of the work on hygienic macros has been in the context of languages without infix operators. Dylan has infix operators, though: https://people.csail.mit.edu/jrb/Projects/dexprs.pdf . Fundamentally, though, infix operators are a feature of read, and read is responsible for handling them.

So if I see this correctly, we cannot substitute syntax objects (since we do not want to hardcode the binding structure of match, and also whether a name resolves to a binder, etc.).

Substitution in syntax objects should be possible, but with variables whose binding is a phase earlier. It makes sense to replace metavariables, but not variables, in a syntax object. The macro hygiene system takes care of making sure that the variables in the final expression point to what they would have pointed to in the original syntax.
I think those syntax examples are very close to what we were talking about. It probably makes sense to give them a bit more structure than just a list of subtems, though. For example, the LHS and RHS of a match clause should probably be grouped together. And because Lean has a fairly rich notation system, a single source span is likely not to be enough. Instead, there should be a source span for the expression as well as a list of tokens that are the keywords and delimiters involved in the expression, with source locations for each, as well as comments and the like.

Notation should probably also remain unexpanded then.

I would think that notations would introduce a new operator for forming syntax objects, a new action for expanding them, and an action for pretty printing the notation.

For linters I think it would be good to have a mapping from syntax objects to the elaborated expressions (so that you can infer types, for example). With such a mapping, we could also do auto-completion and other editor functions in a nice way.

The way it works in Racket is pretty nice. Syntax objects have arbitrary key-value maps associated with them, and one of the properties in that is the original, user-written syntax that eventually gave rise to the current object. In Lean, elab could be syntax -> term, and then the term could say "I'm from there!".

I've become pretty convinced that the expansion process makes the most sense as going directly to terms. Imagine expand : syntax -> elab term, where elab is a monad with effects suitable for expansion-time. I see that as a subset of the tactic effects, including things like introducing a new metavariable, associating syntax objects with that metavariable to be expanded later, doing explicit calls to unification, etc. This allows syntax expansion to be type-directed, and keeping only deterministic operations in elab means that expansion can be done incrementally.

Fundamentally, though, infix operators are a feature of read, and read is responsible for handling them.

Maybe infix operators were a bit of red herring.  What I was wondering is whether there are any good solutions to user-defined syntax:
infix `#` := ... -- declare on line 1
#check 1 # 2 -- and on line 2 read already needs to parse a different grammar

And because Lean has a fairly rich notation system, a single source span is likely not to be enough.

Ah, originally I thought it would be enough since you can always recover the input tokens by rerunning the scanner.  At least that works well enough for scala.meta (on second glance they just discard comments in transformations...).
But that makes modifying (and then pretty-printing) syntax objects awkward.  Ideally you could transform syntax trees by just traversing the tree.  Say you want to rename a constant c to d, then ideally you'd just map over the syntax tree, and replace each occurence of (ident c) with (ident d), and pretty-print the result.  But for that to work we need to be able to print syntax trees including comments, etc., without reference to the original file.
Do you have an idea on how to store that nicely?  Maybe an extra case in syntax?
def tokens := string -- we can expand this

meta inductive syntax
| num (n : ℕ)
| name (n : name)
| string (s : string)
| quote (e : expr)
| node (o : origin) (kind : name) (children : list syntax)
-- new:
| source (tks : tokens)

/-
λ x, x + 2
-/
#eval
let o (l1 c1 l2 c2 : ℕ) := origin.file ⟨"example2.lean", ⟨l1,c1⟩, ⟨l2,c2⟩⟩ in
syntax.node (o 1 0 1 10) `lambda [
    syntax.source "λ ",
    syntax.node (o 1 2 1 3) `ident [
        syntax.source "x ",
        syntax.name `x,
    ],
    syntax.name `default,
    syntax.source "",
    syntax.node origin.synthetic `placeholder [],
    syntax.source ", ",
    syntax.node (o 1 5 1 10) `infix [
        syntax.node (o 1 5 1 6) `ident [
            syntax.source "x",
            syntax.name `x
        ],
        syntax.node (o 1 7 1 8) `ident [
            syntax.source " + ",
            syntax.name `+
        ],
        syntax.node (o 1 9 1 10) `num [
            syntax.source "2",
            syntax.num 2
        ]
    ]
]

The interleaving has the advantage that we can always print back syntax trees, even if they have unknown nodes.  On the other hand, building them by hand is ugly (since you need to provide the tokens in addition to the trees).  So I think this might be the wrong approach.

Syntax objects have arbitrary key-value maps associated with them, and one of the properties in that is the original, user-written syntax that eventually gave rise to the current object.

In Lean, (elaborated) expressions have a tag field (an unsigned integer).  I guess we could keep a map/array of syntax trees after elaboration, and have the tags index into this array.

In Lean, (elaborated) expressions have a tag field (an unsigned integer). I guess we could keep a map/array of syntax trees after elaboration, and have the tags index into this array.

@gebner I think we should replace m_tag with a proper annotation object.
The tag was added in Lean 0.1, when we had completely different ambitions for Lean. At that time, the goal was: a prover backend for dependent type theory. Now, we view Lean as a programming language and a proof assistant. The overhead of having an extra pointer is acceptable. Moreover, it would simply our design. For example, we would be able to remove the pos_info_provider.

@david-christiansen Is there anything better in the literature? Skimming through the papers that Leo posted, nobody even seems to mention infix operators.

@gebner David sent me the following additional reference:
http://www.ccs.neu.edu/home/dherman/research/papers/dissertation.pdf

But that makes modifying (and then pretty-printing) syntax objects awkward. Ideally you could transform syntax trees by just traversing the tree. Say you want to rename a constant c to d, then ideally you'd just map over the syntax tree, and replace each occurence of (ident c) with (ident d), and pretty-print the result. But for that to work we need to be able to print syntax trees including comments, etc., without reference to the original file.

@gebner I think it would be nice to be able to transform syntax objects. That is, implement syntax to syntax transformations. If we want to do this, (I think) we would need explicit binding sites in the syntax object. For example, the notation have h : A, from P, R would be represented by the syntax object
syntax.node `have [(syntax.binder `h [A, P] [R])]

where syntax.binder is a primitive used to encode binding sites. We would also have a de-Bruijn variable construct syntax.var idx. In the node R, h would be represented by syntax.var idx.
Remark: I'm suppressing origin, source, etc.
Similarly, lambda (x : A), B would be encoded as
syntax.node `lambda [(syntax.binder `x [A] [B])]

I discussed this approach with @david-christiansen, and he pointed out that it complicates the reader. We would need to have something like our m_local_decls. Moreover, for some syntactic constructs (e.g., match-expressions) the binding sites are not explicit. Things can get very complicated with we allow an extensible notion of pattern.
Let's call this design "explicit binding sites at syntax" (1). I have a bias towards it because I know how to implement it efficiently, and we would be able to have syntax to syntax transformations.
Another possible design (2) is "no binding sites at syntax". In this design, bindings are resolved at expansion time. That is, at expand : syntax -> elab expr. I have a pretty good idea how to implement this one. However, it is not totally clear how to implement sophisticated syntax to syntax transformations without producing expr as an intermediate step. David suggested the following idiom for transforming s : syntax. First, we expand and obtain e : expr := expand s. e subterms would contain references to s nodes. Then, we would traverse e to figure out the "meaning" of s nodes, and then transform s using this information. I'm not fully convinced this can be done. My main concern is that some of the information may be lost in the translation to e. What do you think?
The main advantage of this design is that read(ers) are much simpler to implement. Binding resolution is implemented once for each kind of syntax node. That is, we don't need to re-implement it for each new reader. Another advantage is that we have more information (e.g., elaborated types) when resolving binding sites.
BTW, I read the paper "Binding as Sets of Scopes" (linked above), and I still don't understand why this approach is better than the locally nameless approach.
In the “Related Work” session, the author says “While our work shares certain goals with techniques for representing resolved bindings, such as de Bruijn indices, higher-order abstract syntax (Pfenning and Elliott 1988), and nominal sets (Pitts 2013), those techniques seem to be missing a dimension that is needed to incrementally resolve bindings introduced and manipulated by macros.”.
I can see the advantages of design (2) (i.e., "no binding sites at syntax"), but I don't see yet the advantages of "Binding as Sets of Scopes". AFAICT, we can implement design 2 by having a local context that maps (user facing) names to local constants in the elab monad. It would be similar to the m_local_decls field we currently use at parsing time.

@kha How do we feel about this change?

I think we also need an expand_command : syntax -> tactic environment.

@gebner I missed this one. I agree with you. However, I think we will also need expand : syntax -> elab expr where elab is a different monad that encapsulates the state of our elaborator.

Maybe infix operators were a bit of red herring. What I was wondering is whether there are any good solutions to user-defined syntax:

I think we should separate elaboration for parsing (i.e., read procedure). In a distant future, we may have multiple readers like Racket, and each of them may provide mechanism for user-defined syntax.
We can start with a reader based on the Pratt's parser we use in the current parser. Note that most of the complication there is because we didn't want to backtrack in the parser.

I'm considering the following approach:
1- reader produces a syntax object similar to the one described by @gebner above. The binding sites are not resolved. So, it is easy to implement different readers.
2- Given an environment, local_context and a syntax object, we resolve binding sites. The result is a syntax object with resolved binding sites. After this step, we would have nodes such as
syntax.node `have [(syntax.binder `h [A, P] [R])]

I'm omitting occurrences, source, etc. The main point here is that binding sites and names have been resolved.
3- The elaborator converts syntax with resolved binding sites into expr.
Note that, we can expand macros and apply other transformations to syntax with resolved binding sites.
We handle this kind of object using the locally nameless approach.
@david-christiansen Does this approach make sense?

syntax.node `have [(syntax.binder `h [A, P] [R])]

Why does syntax.binder take A and P as arguments?  (I'm concerned that we end up having two different syntax representations for before and after name-resolution.)  We could put them into the syntax.node part instead (just like before name-resolution):
syntax.node `have [A,P, syntax.binder `h R]

The explicit binder approach should also work in more interesting cases, such as match:
syntax.node `match [
  A,
  syntax.binder [`x, `y, `z] $ -- or multiple binders?
    syntax.node `equation [
      /- x::y::z::_ -/,
      /- x+3 -/
    ]
]

Just to summarize my thoughts on the binding strategies:

No binding information: Macros will have lots of mk_fresh_name calls.  Syntax extensions need to define substitution and free variable operations.
Binding information introduced by the parser: This requires too much parser state for my taste (I think the parser could otherwise be stateless).  Syntax extensions don't need to define any extra operation, but need to handle binding during parsing.
Extra name-resolution step: Seems like a reasonable compromise. Syntax extensions need to define the name/binding resolution operation.

David suggested the following idiom for transforming s : syntax. First, we expand and obtain e : expr := expand s. e subterms would contain references to s nodes. Then, we would traverse e to figure out the "meaning" of s nodes, and then transform s using this information. I'm not fully convinced this can be done. My main concern is that some of the information may be lost in the translation to e. What do you think?

This issue is independent of the binding representation that we choose.  Just to clarify, I think this idiom is intended to (partly) address queries such as the following:

Given a syntax (sub-)tree, what is its type after elaboration?
Given a syntax (sub-)tree, what is its expression after elaboration?
Given a syntax (sub-)tree, what is the expected type during elaboration?
Given a syntax (sub-)tree, what is the local context during elaboration?
Given a syntax (sub-)tree t and a modified version t' what is the elaborated type/expression of t'?
Given an expression, what was its syntax tree?

Note that these queries are about sub-trees, so you can't just run the elaborator/expand on them.  (I'd like answers to all of these queries.  For autocompletion, 1 and 3 are important.  I don't think these queries would be used much in syntax extensions.  I think these queries would be handy for linters and refactoring.)
In David's approach, we could store the syntax tree in the annotation of the expression.  Then we could go from expressions to syntax trees.  This would allow us to answer queries 1, 2 and 6.

We could put them into the syntax.node part instead (just like before name-resolution):
syntax.node have [A,P, syntax.binder h R]

Great.

Extra name-resolution step: Seems like a reasonable compromise. Syntax extensions need to define the name/binding resolution operation.

Great. I prefer this option too.
The "No binding information" is the worst one in my point of view. Note that we would still have to perform binding site resolution to be able to implement substitution and free variable operations.

In David's approach, we could store the syntax tree in the annotation of the expression. Then we could go from expressions to syntax trees. This would allow us to answer queries 1, 2 and 6.

I think this is great. It simplifies error reporting, and we may also use the information to improve the pretty printer. For example, last week, @jroesch pointed out that the pretty printer has no support for match-expressions, and this may confuse users when visualizing goals.

(I'm still reading through the papers, there was a four-day bridge weekend here in Germany)

I think it would be helpful to consider, from a high level, a few non-trivial procedural metaprograms (what we call "macros" over in Lisp-land) when thinking about how to do binders in syntax.
Imagine I wanted to implement my own pattern matcher and my own embedding of an OO language. A pattern matcher needs to discover which of the identifiers in patterns are references to things like constructors or directives that change the matching, and which are variables to be bound in the right hand side of the matching clause. An OO language has an identifer like self that is bound inside of instance methods, and may also bring the instance's fields into scope in a method.
In each case, scope resolution is pretty non-trivial. The syntax is going to be processed by a fairly complicated metaprogram, and the binding structure is determined by the binding structure of the resulting core-language program, because the core-language program is the meaning of the expression. Having a name-resolution phase prior to expansion means that a large part of each of these nontrivial metaprograms needs to be written twice, once to resolve bindings, and once to emit the realization of that binding structure in the next step.
If the reader is what resolves names in syntax, then the reader needs to be extended to support self in an embedded OO language, and to know which names are constructors. If there is a pass prior to expansion, then a lot of work needs to be duplicated by metaprograms with interesting binding structure.
I think that @gebner missed a strategy, which is essentially the Scheme/Racket strategy: expanding syntax proceeds out->in, and emits bindings as it goes. The expansion monad tracks the context in which a piece of syntax is being expanded, and uses that to resolve identifiers that are emitted by expansion. So in order to implement a binder, a syntax extension expands to a binder. This become de Bruijn indices when the identifier is finally resolved.
How about this for an API sketch?

The reader (parser) maps input strings to syntax objects.
Syntax objects are essentially s-expression-like, with added metadata like source locations and the scopes in which they participate.
There is an expander monad, which has most of the pure operators from tactic. Keeping it pure lets it do things like look up definitions, but it preserves incrementality in editors by allowing unchanged expansion results to be cached.
expander also has a task queue, and an operation: enqueue : syntax -> expander metavar that creates a new metavariable and schedules it to be solved by expanding the syntax object in it.
Metaprograms have the type expander expr, where expr is the kernel language.

Under this kind of API, syntax can straightforwardly expand into new syntax by enqueueing it and emitting a reference to the resulting metavariable. Additionally, the binding structure can be discovered incrementally as the program is expanded, at which point the surrounding context is available for resolving names, and expansion can probably be parallelized to some extent.
What this doesn't provide is substitution and free-variable analysis without expanding syntax. Much of the utility of substitution can be recovered by emitting a let-binding, or by having a meta-level let-binding that doesn't leave a trace in the resulting program. Otherwise, the answer is to first annotate each identifier in the original syntax object with a unique ID, then expand it, and recover the binding structure from the kernel language expression that results.

@david-christiansen Thanks for the feedback. Your design makes sense. My concern is that it doesn't support operations such as expand_macros : syntax -> syntax. That is, we would only be able to expand syntax into expr. One of the papers above says this is an useful operation (e.g., for debugging macros). Is this an issue?
It is also not clear to me whether we will be able to implement robust refactoring tools (i.e., transformations from syntax to syntax). It would be great to have one for renaming declarations. I agree with you that we can inspect the generated expr to find out how names were resolved. My concern here is that information may be lost when translating/expanding into expr. For example, consider the scenario where the generated expr does not contain any sub-expression liking to an identifier in the input syntax. I think this may happen when elaborating complicated declarations. Should we treat this kind of situation as a bug?
Here is an orthogonal issue, one of the paper above stresses the impact of macro expansion on debugging. We will have a similar issue: the impact of macro expansion on proofs. In the paper, they say it is great to preserve as much of the original structure as possible. It is not clear how to do it when proving properties of definitions that make heavy use of macros. Before reading the paper, I was assuming a very naive approach: just expand the macros :) Should we be concerned about this?

Racket's local-expand is certainly highly useful. Not only is it useful for inspecting the output of expansion, it is also useful for inverting control to implement other control structures in metaprograms. There is not a separate expr datatype in the Scheme family, but typically a subset of syntax objects are taken to represent the core forms, and this corresponds to expr in the above.
It is also very useful to be able to single-step expansion, which I think is supported pretty well by giving the current expression and the expansion queue.
If the result of expansion does not have any references to a binding, then by definition there is no reference to worry about, so renaming becomes trivial. What breaks this is when hygiene is bent to introduce new identifiers. For example, defining a datatype introduces new names based on the original one. In Racket, syntax properties are used to communicate this fact to tools. Presumably, an additional constructor would make sense for that as well.
I think that proving properties of definitions that use macros should exactly have the "just expand the macros" semantics. The meaning of any expression can only be the meaning of its elaboration in the kernel language.

the binding structure is determined by the binding structure of the resulting core-language program, because the core-language program is the meaning of the expression.

I think this is a non-sequitur.  We can define a binding structure on syntax trees, and in a fundamental sense I think the user sees it this way as well.
When you have an expression like match x with (y::ys) := y+1 | _ := 42 end, then clearly the y in y+1 is bound.  But I don't think anybody would justify this by saying that the match expands to list.cases_on x ... (λ y ys, y+1).  I think it is much more natural to argue that it is bound because the name occurs on the left hand side of the match case, i.e. by arguing on the level of the syntax tree instead of the elaborated expression.

Having a name-resolution phase prior to expansion means that a large part of each of these nontrivial metaprograms needs to be written twice, once to resolve bindings, and once to emit the realization of that binding structure in the next step.

This is a good point.

Syntax objects are essentially s-expression-like, with added metadata like source locations and the scopes in which they participate.

The participating scopes is basically a list name, right?  And everything in a definition is in the scope?
-- begin of scope
def foo (x : list nat) : nat :=
match x with (y::ys) := y+1 | z := 42 end
-- end of scope

There is an expander monad, which has most of the pure operators from tactic. Keeping it pure lets it do things like look up definitions, but it preserves incrementality in editors by allowing unchanged expansion results to be cached.

I don't think restricting tactic buys us anything.  Caching expansion results is probably less useful than it seems--we can only reuse results if the environment hasn't changed, and most of the time it changes.  (There are further technical issues with caching, see #1601.)

expander also has a task queue, and an operation: enqueue : syntax -> expander metavar that creates a new metavariable and schedules it to be solved by expanding the syntax object in it.

There should probably be an extra argument for the expected type:
enqueue (expected_type : expr) (syn : syntax) : expander expr

I guess the local context is handled as state of the expander monad, and saved when calling enqueue.

Much of the utility of substitution can be recovered by emitting a let-binding, or by having a meta-level let-binding that doesn't leave a trace in the resulting program.

Explicit substitutions are a nice approach for macro expansion.  But they don't really help if you want to get a syntax tree back to insert it in the editor (ideally refactoring would be able to remove let-bindings).

I think I have an idea on how to deal with both queries 1-6 and the problem of renaming bound variables.
We can add a "tracing" mode to the expander/elaborator, where we store all calls to enqueue and add operation to inspect them (or even by default, depends on performance):
get_local_context : syntax -> expander expander_local_context
get_expected_type : syntax -> expander expr
get_metavar : syntax -> expander expr
-- maybe, not sure how easy/reliable this is
find_source : expr -> expander syntax

Then we can implement variable renaming (fairly) reliably by:

expand the source syntax
Find the syntax.identifier node that is at the position in the editor.  Obtain its corresponding local constant by get_metavar >>= instantiate_mvars.
Traverse the source syntax, at every identifier node call get_metavar >>= instantiate_mvars and see if it is the local constant that we want to replace.  If so, rename the identifier node.

(In a sense, we annotate the source syntax with fresh variable names that satisfy the regularity condition: every binder has a different bound variable name.)

It is also very useful to be able to single-step expansion

Does this work well with the sets-of-scopes approach?  If I see it correctly, you could have an eta macro that expands as follows:
-- f : nat -> nat -> nat
λ x, eta (f x)
λ x, λ x, (f x) x

(Each of the x variables in a different scope, one is in the macro's scope, the other one in the definition's.  I hope I got this right.)
Clearly, the expanded version is not the right surface syntax. We also can't directly rename one of the x variables since we don't have binding information (yet).  How do we pretty-print the expanded syntax trees?

The method you mention for renaming seems equivalent to the Racket way of doing it, which is to stick a unique syntax property value on each identifier, expand it, and traverse the result with a parser to figure out which unique ID binds which. Storing a reference to the original syntax of each node in the resulting expr does basically the same thing. But this doesn't require instrumenting the expander.
To pretty print the expansion of the eta macro, use the scope sets to discover what refers to what, rename as needed to remove apparent conflicts, and then print the renamed version. Binding information is available post-expansion.

BTW, I read the paper "Binding as Sets of Scopes" (linked above), and I still don't understand why this approach is better than the locally nameless approach.

I think one advantage is that you can specify macros via just rewrite rules, even if they match on binders.  In pseudo-Lean syntax:
-- not shown: parsing/printing hooks for `where` syntax, and macro boilerplate.
match syn with
| `````(%%a, where %%b) := `````(let %%b in %%a)
end

example := n, where n := 5
example := m, where ⟨m,n⟩ := (⟨1,2⟩ : ℕ×ℕ)

This specification also works transparently with destructuring assignments.  With an explicit name resolution step, we would have to reimplement/reconstruct the bindings that let would create.

Storing a reference to the original syntax of each node in the resulting expr does basically the same thing. But this doesn't require instrumenting the expander.

As Leo remarked, this will not work (reliably).  There are syntax objects that do not show up in the final expression (for example identifiers in binders, i.e. the first x in λ x, x+1).  Besides, the elaboration/expander state would already contain all of that information, so it's just a matter of storing the elaboration state.

To pretty print, [..] use the scope sets [..]. Binding information is available post-expansion.

I feel like this may be a major problem if we view expansion as a transformation from syntax to expr.  The main issue is that we need to execute tactics during expansion/elaboration.  Fundamentally, I don't think there is any way to avoid tactic execution--the result of tactics affects the types of variables in the local context and the expected type.  And we definitively want type-directed expansion: the ⟨_, _⟩ syntax can expand to any structure constructor (prod.mk, sigma.mk, etc.) depending on the expected type.
Tactic execution is relatively expensive (and potentially non-terminating), so I think we should not require it to print syntax objects.

Further points:
Expanding a primitive binder node (think of lambda or let) in the sets-of-scopes approach requires adding a new scope to all identifiers under that node.  In contrast to locally nameless, I don't see any way to optimize this using variable bounds.  I don't know if this would be a performance problem.
In Racket, there is presumably a good way to name global scopes (that is, what module you're in, what module imports and top-level definitions are visible, etc.).  In Lean, we don't have that (and it would essentially require storing the environment at that point due to open, namespace, section, etc.).  (I think storing these environments is a bad idea due to memory usage.)  Hence syntax quotes referencing constants would probably be unhygenic.  With an extra name resolution step, we could resolve the syntax quotes where they are defined, and this problem would not arise.
section
open nat  -- now bodd resolves to nat.bodd
def my_quote := `````(bodd)
end

-- inserting my_quote now would give an "unknown identifier" error,
-- unless we remember the environment my_quote was declared in.

@gebner Thanks for sending the last message. It is very useful.
After I read your message,  I started to question the feasibility of syntax -> syntax transformations. Name resolution is also problematic before elaboration. We have aliases that are solved using type information that is only available during elaboration.
import data.stream
open stream
open list
constant a : list nat
constant b : stream nat
#check cons 1 a  -- cons here is list.cons
#check cons 1 b -- cons here is stream.cons

In the current implementation, we use a choice node to encode overloaded names. We would need to use something similar to be able to resolve names before elaboration. So, any syntax -> syntax transformation would have to take these choice nodes into account. BTW, the reader will also have to introduce choice nodes for overloaded notation.

As Leo remarked, this will not work (reliably). There are syntax objects that do not show up in the final expression (for example identifiers in binders, i.e. the first x in λ x, x+1). Besides, the elaboration/expander state would already contain all of that information, so it's just a matter of storing the elaboration state.

I agree, but what do we do then? We both agree this approach will not work reliably.
Resolving names before elaboration and overloaded notation also seem to create problems for syntax -> syntax transformations.
BTW, I'm also concerned about printing syntax objects reliably after we apply transformations.
Even a simple refactoring tool that just renames declarations will be tricky to work reliably and produce nice results. A rename may introduce an unintended overloading due to aliasing, and the new syntax may fail to be elaborated. The refactoring tool would need a mechanism to decide when a fully qualified name is required or not. Some conservative solutions may be sufficient (e.g., it uses a fully qualified name if the short name is overloaded).

I think one advantage is that you can specify macros via just rewrite rules, even if they match on binders. In pseudo-Lean syntax:

I have a question about this approach. Suppose we define bor a b to be a macro defined as
(if x then x else b) where x := a

Now suppose we want to expand bor at let x := tt in bor ff x, we need to know that where is introducing a binding site. Otherwise, we will get the incorrect result let x := tt in (if x then x else x) where x := ff.

After I read your message, I started to question the feasibility of syntax -> syntax transformations.

So the alternative is to expand everything directly into expressions, without being to use syntax as an intermediate step?
This means e.g. you cannot expand into do-notation or even >>=-applications.  You would have to create the fully elaborated expressions directly (including monad type, intermediate result types, the monad instance, etc.)  You'd also need to handle the propagation of the expected type manually.
Even if neither the LN nor the sets-of-scopes approach works out, we can always fall back to obscene amounts of mk_fresh_name.

Name resolution is also problematic before elaboration. We have aliases that are solved using type information that is only available during elaboration.

Yeah, we would need choice nodes in resolved syntax.  The important part about resolved syntax is that local bindings are resolved, otherwise we can't do any transformations/printing at all.

BTW, the reader will also have to introduce choice nodes for overloaded notation.

My plan was to have (parser) notation as a separate node.  That is, basically just store the tokens+expressions in a syntax node, and resolve them during elaboration.  For pretty-printing we want the original (notation) structure anyhow.

Resolving names before elaboration and overloaded notation also seem to create problems for syntax -> syntax transformations.

So far, the name-resolution approach seems to be the least problematic IMO.  I don't think choice nodes are a huge problem, there is tons of other syntax which resolves to different constants depending on the type (structures, calc, brackets, ...).

A rename may introduce an unintended overloading due to aliasing, and the new syntax may fail to be elaborated.

You mean, like renaming stream.cons to scons and then correctly renaming only half of the cons occurrences to scons?  All refactoring tools have limits, everybody expects them to break in complicated situations.  I think many refactoring tools don't even handle capture-avoiding renaming.  Even IDEs such as IntelliJ will tell you "no, I can't do that" sometimes.
Practically speaking, we can improve the accuracy of such transformations by looking at the resulting expression after elaboration.  IMHO the main purpose of the name resolution step should be to separate local bindings from global ones, and determine the local binding structure.

The refactoring tool would need a mechanism to decide when a fully qualified name is required or not.

Essentially, yes.  I don't think it will be a big problem in practice, we can remember and restore the original constant name, and pray that it still works.
I think this part of the refactoring tool should be a general function that is also used to implement pp.  That is, I think pp should be implemented in a few separate steps:

unexpand : expr -> syntax: leaves out implicit arguments, class instances, tries to introduce notation, do-notation, structures, projections, brackets, etc.  Returns resolved syntax.
unresolve : syntax -> syntax: replaces LN-bound variables by ident nodes, in such a way that identifiers do not clash.
print : syntax -> format: turns the unresolved syntax object into a string

The refactoring tool would only use print ∘ unresolve, and that should be reasonably safe in most scenarios.

I have a question about this approach.

The trick is that each occurrence of x has a set of scopes attached, in this sense the two xs are different. Let's trace the calls to expand.  After the first step, we invent the new scope 0 for the primitive binder let, add x^{0} to the local context, add the scope 0 to all identifiers in the body, and proceed with the subterm.
let x^{} := tt in bor ff x^{}
bor ff x^{0}
(if x^{1} then x^{1} else x^{0}) where x^{1} := ff
let x^{1} := ff in if x^{1} then x^{1} else x^{0}
if x^{1,2} then x^{1,2} else x^{0,2}

If I read the paper correctly, then the x^{0,2} at the end resolves to x^{0} instead of x^{1}, because {0} is the (largest) subset of {0,2}.

So the alternative is to expand everything directly into expressions, without being to use syntax as an intermediate step?

It is not clear to me what is a good alternative. It seems each approach has problems/limitations.
I'm looking forward to meet you in Cambridge and discuss these issues in person.

You mean, like renaming stream.cons to scons and then correctly renaming only half of the cons occurrences to scons?

Yes, I'm thinking of examples like this. I'm also considering the case where scons is aliased.
That is, there is a foo.scons, and we have used open foo.

All refactoring tools have limits, everybody expects them to break in complicated situations. I think many refactoring tools don't even handle capture-avoiding renaming. Even IDEs such as IntelliJ will tell you "no, I can't do that" sometimes.

I'm ok if the refactoring tool has limitations, but I want to understand better what we will not be able to do before we decide which approach we will use.
BTW, I'm also concerned with hole commands. After this refactoring, they will return syntax objects representing different ways of filling a hole.
Some of these fragments will fail to elaborate due to overloading, missing implicit arguments, etc.
At least, in the new approach, we can try to re-elaborate these fragments before we send them to the user.

The trick is that each occurrence of x has a set of scopes attached, in this sense the two xs are different. Let's trace the calls to expand. After the first step, we invent the new scope 0 for the primitive binder let, add x^{0} to the local context, add the scope 0 to all identifiers in the body, and proceed with the subterm.

How do we know that the where is creating a scope?
That is, when we go from
bor ff x^{0}

to
(if x^{1} then x^{1} else x^{0}) where x^{1} := ff

How do we know that the where is creating a scope?

Let me explain where each of the scopes come from / what they mean:

0: scope introduced by the let-binding that the user wrote
1: scope where the macro was defined
2: scope introduced by the let-binding of the macro expansion

(Maybe there should be another scope for the current definition, the paper is not too clear on the total number of scopes in the actual implementation.)
The beauty of the SOS approach is that expanding where does not introduce a scope.  We never need to know the binding structure of macros, we just expand blindly.  The scope only gets introduced when we process the let.
Note that this only works because we never look at subterms of unexpanded terms.  Expansion only happens at the top-level, we only proceed to subterms of primitive constructs (let, lambda, etc.).  There we know the binding structure and introduce scopes as needed.

Some idle thoughts:
syntax.node should take a tag (of type nat) just like expr does now.  Otherwise it becomes difficult to trace syntax object through the elaboration process.  (Just think of my example query: what is the type after elaboration?  If the syntax object is under a binder, we'll instantiate it with different local constants (or scopes) every time, so we can't just use the syntax object itself as an identifier.)
After listening to Jeremy's talk today I remembered that we have another source of unhygiene, namely unresolved expressions in tactics:
-- say bor is a macro such that:
-- bor a b   ==>   let x := a in if x then x else b

example (x : nat) := bor ff (by exact x > 0)

-- expands to:
example (x : nat) :=
let x := ff in if x then x else (by exact x > 0)
                          -- ideally this ^ would still resolve to the parameter

example (x : nat) :=
let x := ff in if x then x else (by exact x > 0)
                          -- ideally this ^ would still resolve to the parameter

Right now, we elaborate x > 0 with respect to the current local context at tactic execution time. Moreover, name resolution is performed using user-facing names. So x will refer to x := ff :(
One possible hack is to make sure that binders introduced by macro expansion use a disjoint set of user-facing names (e.g., they have the prefix _m).
What do you think? Do you have cleaner solutions in mind? Recall that the following weird example is expected to work (and it does in the current version).
example (x : nat) (y : bool) : bool :=
begin
  rename y x,
  exact bnot x -- works, x is now referring to `y : bool`.
end

One possible hack is to make sure that binders introduced by macro expansion use a disjoint set of user-facing names (e.g., they have the prefix _m).

That's exactly what I had in mind.  We already use underscore as a prefix for such private names, and I think that one way or another, macro expansions should just get their bound names prefixed by an underscore.  I see two possible points where this can happen:

Provide an "extra-hygienic" syntax quote, e.g. quadruple-backticks preserves bound names, quintuple-backticks prefixes all bound names with underscores.
Prefix the bound names during substitution of the macro replacement.

When constructing syntax objects manually, the underscores need to be added manually.
The underscores pose no problem for pretty-printing (e.g., when expanding a macro in the editor), the unresolve function can remove the underscore to produce names that we can parse again.

Another option is to remember the local context of the begin-end block during name resolution.  At tactic runtime we would prefix all user-facing names of local constants that were not present during name resolution with an underscore.
(A similar approach is possible with SOS: we would remember the scope set of the begin-end block during expansion, and then use the same scope set when expanding x > 0)

By popular demand over dinner: here is the mockup code.


gcc_plugin crash

this happens in a lot of larger projects when compiling with the gcc_plugin - afl-gcc-fast/afl-g++-fast :
[+] Instrumented 11 locations in TIFFInitJPEG
during GIMPLE pass: evrp
tif_jpeg.c: In function ‘JPEGFixupTagsSubsamplingSec’:
tif_jpeg.c:2388:1: internal compiler error: Segmentation fault
 }
 ^
0x7ffff758e83f ???
	/build/glibc-vjB4T1/glibc-2.28/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
0x7ffff757b09a __libc_start_main
	../csu/libc-start.c:308
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <file:///usr/share/doc/gcc-8/README.Bugs> for instructions.
make[2]: *** [Makefile:696: tif_jpeg.lo] Error 1

to reproduce:
wget http://download.osgeo.org/libtiff/tiff-4.0.4.tar.gz
tar xzf tiff-4.0.4.tar.gz
cd tiff-4.0.4
CC=afl-gcc-fast CXX=afl-g++-fast ./configure --disable-shared
make

Its always the
#include "tiffiop.h"
that causes the crashes for me.
If i comment that out, the compile pass does not crash, but fails to
link later...
Strange.
Heiko
On 7/20/19 9:38 AM, van Hauser wrote:

this happens in a lot of larger projects when compiling with the
gcc_plugin - afl-gcc-fast/afl-g++-fast :
|[+] Instrumented 11 locations in TIFFInitJPEG during GIMPLE pass:
evrp tif_jpeg.c: In function ‘JPEGFixupTagsSubsamplingSec’:
tif_jpeg.c:2388:1: internal compiler error: Segmentation fault } ^
0x7ffff758e83f ???
/build/glibc-vjB4T1/glibc-2.28/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
0x7ffff757b09a __libc_start_main ../csu/libc-start.c:308 Please submit
a full bug report, with preprocessed source if appropriate. Please
include the complete backtrace with any bug report. See
<file:///usr/share/doc/gcc-8/README.Bugs> for instructions. make[2]:
*** [Makefile:696: tif_jpeg.lo] Error 1 |
to reproduce:
|wget http://download.osgeo.org/libtiff/tiff-4.0.4.tar.gz tar xzf
tiff-4.0.4.tar.gz cd tiff-4.0.4 CC=afl-gcc-fast CXX=afl-g++-fast
./configure --disable-shared make |
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/vanhauser-thc/AFLplusplus/issues/25?email_source=notifications&email_token=AB6UEZBJYVJUHFDLQK2J23LQAK6I5A5CNFSM4IFN2J7KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HANQAVA,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB6UEZFKFFHGWG3BMUZTZNTQAK6I5ANCNFSM4IFN2J7A.

I have trimmed down in tif_packbits.c one case of

GIMPLE pass switchconv

crash to this meaningless piece of code:
int
PackBitsEncode(signed long cc)
{
 enum { LITERAL, LITERAL_RUN } state;
 while (cc > 0) {
  for (; cc > 0 && cc % 42; )
   cc++;
 again:
  switch (state) {
  case LITERAL:
    if (cc > 128) {
     goto again;
    }
   break;
  case LITERAL_RUN:
   goto again;
  }
 }
 return (1);
}

The 10th instrumentation of 11 locations produces the crash.

also if optimization is disabled it does not break: AFL_DONT_OPTIMIZE=1

gcc_plugin is inefficient and broken and needs to be rewritten anyway. issue closed

The crash seems to be fixed by https://github.com/vanhauser-thc/AFLplusplus/commit/4ce57e978fd2d8a6f572d7d06761b31644e3c9c8 now. A lot of TODOs left...


When posting as JSON, null goes as "None"

When you post an object as json to an endpoint for a type hug.types.text, such as:
{"x": null}
The value of "x" is received as str "None". Should have been NoneType None instead.

This has been resolved in release 2.2.0, which was pushed to PYPI just a few days ago (after this bug was reported)
Thanks!
~Timothy


Global cache and system file list

version: restic 0.9.5 compiled with go1.13beta1 on linux/amd64
What should restic do differently? Which functionality do you think we should add?
Currently started backup and stopped it many times, to exclude mainly caches. With excluding sysfiles, you could back up 'everything' (root of the system or C:)
2 new flags: --exclude-known-caches and --exclude-sysfiles
They are 2 program-global (built in) --exclude-file lists.

It would probably only make sense, if this list changes, depending on the operating system.

Instead of making our own local lists, make collaborative lists, saving time.

Ideally this would be built in and on by default.

I think it is best to let this be off by default, until we feel more confident.

By built in, it's probably better to have it independent of the binary, download the list from github, check integrity, cache the file (say get it every month).

How do you verify, that the list actually comes from Restic?

Have --online-exclude-list https://something?

What are you trying to do?
Make Restic easier to get started and free up backup storage.
Did restic help you today? Did it make you happy in any way?
Decided to start hourly backups until I get up better infra.

@jtagcat Nice thinking to consolidate many people's efforts that are probably spread out and very redundant. I agree that it's unlikely that this would be a thing to maintain in the restic project itself, it's better that it's a collaborative resource in the community.
Have you seen https://restic.readthedocs.io/en/latest/040_backup.html#excluding-files ? I think that the existing exclude flags supports what's needed.
Imagine you have a couple of files from the community, collaboratively built to contain excludes for your OS and what not. You can simply download it and start using it with these existing exclude flags. It would be similar to what people are already doing with dotfiles and IDE exclusion lists, there's a lot of this going on already.
So, I'd say the real suggestion is to agree on where to keep such lists. The forum could be one place.

I'd propose github.com/restic/exclude-lists
As I'm planning to set up quite a large operation of backups and have decided on restic :tada: I'm willing to be the maintainer of the repository. So I'd ask @fd0 to create the repository.
As far as integrating the lists, I'd say first make the lists, see how people are using them, test, test, test, create stable, beta, nightly (or equivelant) branches and then even think of direct integration.

In case it happens, I'd suggest github.com/restic/lists instead. It's be impractical to make the repo specific to exclude lists, in case restic gets include support later on (something that's been requested). Let's always try to think ahead when deciding on naming and stuff.
It should also be made clear in the repository that the data in there is community driven and not kept up to date by the restic project.

@jtagcat Have you started those lists yet? Can you put them in the Recipes section in the forum? I'm inclined to close this issue, because what we're discussing here is not something that will currently be maintained in the restic repository. But it would be a great addition to the community in that Recipes section, it's exactly what it's there for!

@rawtaz https://github.com/jtagcat/restic-lists
Will do!

Why not just using github/gitignore?

Linux
Windows


Set speed limit for gps103 protocol

Is it in your plans to create a new command that will be possible to set up speedlimit on gps103 protocol devices.
The command that is suposed to be sent is
H,SPEED_LIMIT_PARAMETER with 3 bytes
For example:
to set up the speedlimit to 80km/h, send to the tracker the command H,080
to set up the speedlimit to 100km/h, send to the tracker the command H,100
to set up the speedlimit to 5km/h, send to the tracker the command H,005
or also set the speedlimit in the device parameter and when a position record comes from the device, trigger a routine to compare the speed sent with the parameter, if it is over limit, send notifications.
Speed limit notifications are very important!!!!!

Are you sure the command is the same for all GPS103 devices?

By the way, you can already send it as a custom command.

All gps103 devices that i have here accepts this command... Maybe it is a
standard...
Em 19 de nov de 2016 22:29, "Anton Tananaev"<EMAIL_ADDRESS>escreveu:

Are you sure the command is the same for all GPS103 devices?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tananaev/traccar/issues/2567#issuecomment-261749497,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ARobDgfv8GwSJj5-nSxUSPXEh3xIeuMGks5q_5RugaJpZM4K3aPE
.

I'll try to send this command as a custom command and keep your updated...
Em 19 de nov de 2016 22:30, "Anton Tananaev"<EMAIL_ADDRESS>escreveu:

By the way, you can already send it as a custom command.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tananaev/traccar/issues/2567#issuecomment-261749527,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ARobDpKAhV13kyvGCK9UubbxRapcqfZfks5q_5SKgaJpZM4K3aPE
.


Feature: remember kendra external incides when using the creation wizard

Issue #, if available:
Description of changes:
Running npm run create a second time will remember the configured External Kendra indices, if any.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

This PR  is to merge to dev. Do you meant to send it for main instead?